diff options
| author | Maxim Kuvyrkov <mkuvyrkov@ispras.ru> | 2007-06-06 15:28:10 +0000 |
|---|---|---|
| committer | Maxim Kuvyrkov <mkuvyrkov@ispras.ru> | 2007-06-06 15:28:10 +0000 |
| commit | e5c3cae9778837598574144265b597f12177154c (patch) | |
| tree | 98491ea20a04a4ff08ead48e54c30577ffb9bbde | |
| parent | d7b13fb3b1ac6545c4fb2d32908c7a9621ce0e50 (diff) | |
Update various pieces of infrastructure.
Implement initial support for ia64 data and control speculation.
* sched-ebb.c (n_insns, sched_n_insns): Rename to rgn_n_insns and
sched_rgn_n_insns respectfully. Update all uses.
(ebb_common_sched_info): Move initialization to schedule_ebbs ().
(add_remove_insn, add_block1, fix_recovery_cfg, ebb_head_or_leaf_p):
Rename to ebb_add_remove_insn (), ebb_add_block (),
ebb_fix_recovery_cfg (), ebb_region_head_or_leaf_p () respectfully.
Update all uses.
* target.h (struct gcc_target: needs_block_p): Change signature.
Update all uses.
(struct gcc_target: gen_check): Rename to gen_spec_check. Change
signature. Update all uses.
(struct gcc_target: get_insn_spec_ds, get_insn_checked_ds):
New hooks to query speculation information of instruction.
(struct gcc_target: skip_rtx_p): New hook to identify speculative
markers of instruction.
* rtlanal.c (may_trap_p_1): Skip through UNSPEC on request of target.
* haifa-sched.c (vecprim.h): New include.
(sched_max_uid, sched_last_basic_block, sched_bbs, sched_insns):
Remove.
(spec_info): Make global.
(sched_init, sched_finish, haifa_sched_init, haifa_sched_finish):
Restructure initialization.
(haifa_speculate_insn): New static function.
(try_ready): Use it instead of speculate_insn ().
(sched_extend_ready_list, sched_finish_ready_list): New functions.
(extend_ready, haifa_local_init, sched_local_init, sched_insns_init):
Remove.
(sched_insns_finish, extend_all): Ditto.
(haifa_insns_init): Rename to haifa_insns_init. Update all uses.
(haifa_luid_for_non_insn): New static function.
(dep_weak): Move to sched-deps.c. Rename to ds_weak ().
(init_before_recovery): Use haifa_init_only_bb () instead of
add_block ().
(create_check_block_twin): Update to use new initialization functions.
(change_pattern): Rename to sched_change_pattern (). Make global.
Split haifa-specific functionality into ...
(haifa_change_pattern): New static function.
(speculate_insn): Rename to sched_speculate_insn (). Make global.
Split haifa-specific functionality into ...
(haifa_speculate_insn): New static function.
(sched_bbs_init, add_block, glat_init_1, glat_init, sched_bbs_finish):
Remove.
(debug_spec_status): Use get_dep_weak_1 () instead of get_dep_weak ()
to prevent ICEs on malformed input.
(extend_bb, init_bb, extend_insn, init_insn, init_insns_in_bb): New
static functions for walking through scheduling region.
(sched_scan): New function for walking through scheduling region.
(sched_extend_bb, sched_init_bb): New static functions.
(sched_init_bbs, sched_finish_bbs): New functions to init / finalize
basic block information.
(attach_life_info1, attach_life_info): Move to the better place.
(sched_luids): New vector variable to replace uid_to_luid.
(sched_luid_extend): Rename to luids_extend_insn (). Update to use
sched_luids.
(sched_max_luid): New variable.
(sched_luid_init): Rename to luids_init_insn ().
(sched_init_luids, sched_finish_luids): New functions to
init / finalize logical uids.
(sched_luid_finish): Remove.
(sched_insn_extend, sched_insn_init, sched_insn_finish): Remove.
(sched_extend_target): New function.
(h_i_d): Make it a vector.
(extend_h_i_d, init_h_i_d, haifa_init_h_i_d, haifa_finish_h_i_d):
New functions to initialize / finalize haifa instruction data.
(haifa_init_insn): New static function.
(sched_bb_extend, sched_bb_init): Remove.
(haifa_init_only_bb): New static function.
(sched_data_update, sched_data_finish): Remove.
* modulo-sched.c (sms_common_sched_info): Move initialization to ...
(setup_sched_infos): New static function.
(sms_schedule): Move initialization of common_sched_info,
sched_deps_info, current_sched_info to setup_sched_infos ().
* sel-sched.c (reset_sched_cycles_p): New variable.
(old_ready_veclen): Rename to max_issue_size.
(substitute_rhs): Rename to substitute_reg_in_rhs (). Update to use
vinsns properly.
(replace_in_vinsn_using_bitmask): Rename to
replace_in_rtx_using_bitmask (). Update to use vinsns properly.
(un_substitute): Update to use vinsns properly.
(replace_src_with_reg): Rename to create_insn_rtx_with_rhs. Don't use
vinsns.
(replace_dest_with_reg_in_vinsn): Rename to create_insn_rtx_with_lhs.
Don't use vinsns.
(replace_dest_with_reg_in_rhs): Update.
(expr_dest_reg): New static function.
(rhs_dest_regno): Use expr_dest_reg ().
(mark_unavailable_hard_regs): Update.
(choose_best_reg): Add assertion. Move logic to ...
(choose_best_reg_1): New static function.
(choose_best_pseudo_reg): Add output parameter.
(find_best_reg_for_rhs): Change type of return value. Update.
(sel_speculation_p): New static variable.
(can_overcome_dep_p, create_speculation_check_insn_rtx): New static
functions.
(apply_spec_to_expr, speculate_expr, has_spec_dependence_p): Ditto.
(un_speculate): Ditto.
(moveup_rhs): Add support for data and control speculation. Update.
(moveup_set_rhs, equal_after_moveup_path_p, compute_av_set): Update.
(propagate_lv_set, compute_live, update_data_sets): Update.
(get_spec_check_type_for_insn): New static function.
(find_used_regs_1, find_used_regs, sel_rank_for_schedule): Update to
support speculation.
(fill_ready_list, find_best_rhs_and_reg_that_fits): Ditto.
(gen_insn_from_expr_after): New static function.
(generate_bookkeeping_insn, fill_insns): Update
(get_dest_reg_from_orig_ops): New static function.
(move_op): Update to support speculation.
(init_seqno): Add parameter. Update.
(sel_restore_other_notes): Move to sel-sched-ir.c.
(is_loop_preheader_p): Move to sel-sched-ir.c. Rename to
sel_is_loop_preheader_p ().
(sel_remove_loop_preheader): Move to sel-sched-ir.c.
(sel_region_init, sel_region_finish, sel_sched_region_1): Update.
(sel_global_init, sel_global_finish): Update.
(selective_scheduling_run): Add .dot dumping.
(handle_sel_sched): Use debugging parameter to choose scheduler.
* sel-sched-ir.c (sel_bb_info): Change array to vector.
(sel_max_uid, lvs, lvs_size): Remove.
(s_i_r_d): New static vector variable.
(get_nop_from_pool): Rewrite.
(free_nop_pool): Update.
(vinsn_equal_insn_p): Remove.
(vinsn_separable_p): Rename to lhs_and_rhs_separable_p. Update.
(vinsn_init): Update.
(sel_insn_rtx_cost): New static function.
(sel_vinsn_cost, sel_gen_insn_from_rtx_after): New functions.
sel_gen_insn_from_expr_after): Ditto.
(rhs_equal_p): Rename to vinsn_correlate_as_rhses_p (). Update.
(rhs_init): Rename to init_expr (). Update.
(rhs_copy): Rename to copy_expr ().
(merge_expr_data): New function.
(rhs_merge): Rename to merge_expr ().
(rhs_equal_insn_p): Remove.
(clear_expr): New function.
(av_set_lookup_rhs): Rename to av_set_lookup (). Update.
(av_set_loopup_other_equiv_rhs): Update.
(av_set_remove_rhs_with_insn): Remove.
(av_set_add_vinsn): Rename to av_set_add (). Update.
(av_set_copy): Update.
(av_set_lookup_insn): Remove.
(av_set_add_insn): Remove.
(deps_init_id_start_insn, deps_init_id_finish_insn): Update.
(deps_init_id_start_lhs, deps_init_id_finish_lhs): Ditto.
(deps_init_id_start_rhs, deps_init_id_finish_rhs): Ditto.
(deps_init_id_reset_deps_to_insn): Rename to
deps_init_id_downgrade_to_use ().
(deps_init_id_note_reg_set, deps_init_id_note_reg_clobber): Ditto.
(deps_init_id_note_reg_use, deps_init_id_note_mem_dep): Ditto.
(sel_cfg_note_p): New static function.
(init_global_and_expr_for_insn): New static function.
(sel_init_global_and_expr): New function.
(finish_global_and_expr_insn_1, finish_global_and_expr_insn_1): New
static functions.
(sel_finish_global_and_expr): New function.
(sel_deps_*): Rename functions to has_dependence_*. Update.
(save_deps_info): Remove.
(setup_has_dependence_sched_deps_info): New static function.
(has_dependence_p): Update.
(tick_check_dep_with_dw, tick_check_p): Update.
(lhs_equals_reg_p): Rename to lhs_of_insn_equals_to_reg_p ().
(get_vinsn_type_for_insn): Remove.
(insn_valid_p): Rename to insn_rtx_valid.
(sel_insn_deffered_init): Remove.
(copy_insn_out_of_stream, copy_insn_and_insert_before): Remove.
(set_insn_init): New function.
(init_insn, init_simplejump, insn_init_move_lv_set_if_bb_header): New
static functions.
(sel_init_new_insns, sel_finish_new_insns): New functions.
(dfa_cost): Rename to vinsn_dfa_cost ().
(bb_header_p): Rename to sel_bb_header_p ().
(bb_empty_p): Rename to sel_bb_empty_p ().
(sel_insn_has_single_succ_p): New function.
(sel_add_or_remove_bb): Update.
(sel_create_basic_block_before, sel_merge_blocks): New function.
(create_insn_rtx_from_pattern_1): New static function.
(create_insn_rtx_from_pattern, create_vinsn_from_insn_rtx): New
functions.
(create_copy_of_insn_rtx, change_vinsn_in_expr): Ditto.
* sel-sched-ir.h (struct _rhs): Rename to 'struct _expr'.
(struct _expr: sched_times, spec_done_ds, spec_to_check_ds): New
fields.
(struct vinsn_def: sched_cycle, sched_times, separable): Remove fields.
(struct vinsn_def: cost, may_trap_p): New fields.
(struct sel_insn_data: sched_cycle): New field.
(struct _sel_insn_rtx_data): New type.
* sel-sched-dump.c (dump_insn_rtx, debug_insn_rtx): New functions.
(dump_vinsn, debug_vinsn, dump_expr, debug_expr): Ditto.
(dump_insn, debug_insn): Ditto.
(sel_dump_cfg_2): Update.
* sel-sched-dump.h (enum _dump_insn_rtx, enum _dump_idata): New enums.
(enum _dump_vinsn, enum _dump_expr, enum _dump_insn): New enums.
* emit-rtl.c (emit_insn_after_1): Call hook.
* sched-deps.c (note_reg_use, note_reg_set, note_reg_clobber): Convert
macros to functions.
(note_mem_dep, note_dep): Ditto.
(sched_analyze_2): Generate control speculative dependencies.
(deps_analyze_insn): Update.
(d_i_d): Convert array to vector.
(deps_extend_d_i_d, deps_finish_d_i_d): New functions.
(sched_deps_local_init): Update.
(get_dep_weak): Move logic to ...
(get_dep_weak_1): New function.
(ds_merge): Move logic to ...
(ds_merge_1): New static function.
(ds_full_merge, ds_max_merge, ds_get_speculation_types): New functions.
(ds_get_max_dep_weak): New function.
* sched-deps.h (sched-int.h): New include.
(struct deps_insn_data: depend): Move field to
'struct haifa_deps_insn_data'.
* target-def.h (TARGET_SCHED_GEN_CHECK): Rename to
TARGET_SCHED_GEN_SPEC_CHECK.
(TARGET_SCHED_GET_INSN_SPEC_DS, TARGET_SCHED_GET_INSN_CHECKED_DS):
New macros.
(TARGET_SCHED_SKIP_RTX_P): New macro.
* sched-int.h (vecprim.h): New include.
(bb_vec_t, insn_vec_t): New typedefs.
(struct sched_scan_info_def): New type.
(sched_scan_info): Declare variable.
(sched_scan, sched_init_bbs, sched_finish_bbs, sched_init_luids):
Declare functions.
(sched_finish_luids, sched_extend_target, haifa_init_h_i_d): Ditto.
(haifa_finish_h_i_d, haifa_init_only_bb): Ditto.
(struct common_sched_info_def): Remove fields which_luid, remove_notes,
bb_extend, bb_init, bb_finish, insn_extend, insn_init. Add field
luid_for_new_insn.
(struct haifa_insn_data): Add fields reg_weight, priority_known.
* Makefile.in (SCHED_INT_H, SCHED_DEPS_H): Update.
(fwprop.o): Fix dependencies.
(haifa-sched.o, sched-deps.o, sel-sched.o, sel-sched-ir.o): Update
dependencies.
* sched-rgn.c (find_single_block_region, haifa_find_rgns): Update.
(extend_rgns, schedule_region, sched_rgn_finish): Ditto.
(sched_rgn_local_preinit, sched_rgn_local_init): Ditto.
(rgn_setup_common_sched_info, rgn_setup_sched_infos): New function.
* sched-vis.c (dump_insn_slim_1): Update.
* config/ia64/ia64.opt (msched-ldc): Rename to msched-spec-ldc.
(msched-control-ldc): Rename to msched-spec-control-ldc.
(msel-sched-data-spec, msel-sched-control-spec): New flags.
(msel-sched-dont-check-control-spec): New flag.
* config/ia64/ia64.c: Rewrite speculation hooks.
* config/ia64/ia64.md (speculable1, speculable2): New attributes.
* params.def (PARAM_SELSCHED_MAX_SCHED_TIMES): New parameter.
(PARAM_ALLOW_START, PARAM_ALLOW_STOP, PARAM_ALLOW_P): Remove.
(PARAM_SEL1_START, PARAM_SEL1_STOP, PARAM_SEL1_P): New parameters.
(PARAM_SEL2_START, PARAM_SEL2_STOP, PARAM_SEL2_P): Ditto.
git-svn-id: https://gcc.gnu.org/svn/gcc/branches/sel-sched-branch@125493 138bc75d-0d04-0410-961f-82ee72b054a4
| -rw-r--r-- | gcc/ChangeLog.sel-sched | 272 | ||||
| -rw-r--r-- | gcc/Makefile.in | 22 | ||||
| -rw-r--r-- | gcc/config/ia64/ia64.c | 880 | ||||
| -rw-r--r-- | gcc/config/ia64/ia64.md | 146 | ||||
| -rw-r--r-- | gcc/config/ia64/ia64.opt | 20 | ||||
| -rw-r--r-- | gcc/emit-rtl.c | 4 | ||||
| -rw-r--r-- | gcc/haifa-sched.c | 1241 | ||||
| -rw-r--r-- | gcc/modulo-sched.c | 26 | ||||
| -rw-r--r-- | gcc/params.def | 32 | ||||
| -rw-r--r-- | gcc/rtlanal.c | 7 | ||||
| -rw-r--r-- | gcc/sched-deps.c | 286 | ||||
| -rw-r--r-- | gcc/sched-deps.h | 49 | ||||
| -rw-r--r-- | gcc/sched-ebb.c | 83 | ||||
| -rw-r--r-- | gcc/sched-int.h | 181 | ||||
| -rw-r--r-- | gcc/sched-rgn.c | 287 | ||||
| -rw-r--r-- | gcc/sched-rgn.h | 16 | ||||
| -rw-r--r-- | gcc/sched-vis.c | 2 | ||||
| -rw-r--r-- | gcc/sel-sched-dump.c | 377 | ||||
| -rw-r--r-- | gcc/sel-sched-dump.h | 90 | ||||
| -rw-r--r-- | gcc/sel-sched-ir.c | 2642 | ||||
| -rw-r--r-- | gcc/sel-sched-ir.h | 455 | ||||
| -rw-r--r-- | gcc/sel-sched.c | 2338 | ||||
| -rw-r--r-- | gcc/target-def.h | 12 | ||||
| -rw-r--r-- | gcc/target.h | 17 |
24 files changed, 5831 insertions, 3654 deletions
diff --git a/gcc/ChangeLog.sel-sched b/gcc/ChangeLog.sel-sched index 82d014ee6ba..21b6b48ab7f 100644 --- a/gcc/ChangeLog.sel-sched +++ b/gcc/ChangeLog.sel-sched @@ -1,3 +1,275 @@ +2007-06-06 Maxim Kuvyrkov <mkuvyrkov@ispras.ru> + + Update various pieces of infrastructure. + Implement initial support for ia64 data and control speculation. + + * sched-ebb.c (n_insns, sched_n_insns): Rename to rgn_n_insns and + sched_rgn_n_insns respectfully. Update all uses. + (ebb_common_sched_info): Move initialization to schedule_ebbs (). + (add_remove_insn, add_block1, fix_recovery_cfg, ebb_head_or_leaf_p): + Rename to ebb_add_remove_insn (), ebb_add_block (), + ebb_fix_recovery_cfg (), ebb_region_head_or_leaf_p () respectfully. + Update all uses. + + * target.h (struct gcc_target: needs_block_p): Change signature. + Update all uses. + (struct gcc_target: gen_check): Rename to gen_spec_check. Change + signature. Update all uses. + (struct gcc_target: get_insn_spec_ds, get_insn_checked_ds): + New hooks to query speculation information of instruction. + (struct gcc_target: skip_rtx_p): New hook to identify speculative + markers of instruction. + + * rtlanal.c (may_trap_p_1): Skip through UNSPEC on request of target. + + * haifa-sched.c (vecprim.h): New include. + (sched_max_uid, sched_last_basic_block, sched_bbs, sched_insns): + Remove. + (spec_info): Make global. + (sched_init, sched_finish, haifa_sched_init, haifa_sched_finish): + Restructure initialization. + (haifa_speculate_insn): New static function. + (try_ready): Use it instead of speculate_insn (). + (sched_extend_ready_list, sched_finish_ready_list): New functions. + (extend_ready, haifa_local_init, sched_local_init, sched_insns_init): + Remove. + (sched_insns_finish, extend_all): Ditto. + (haifa_insns_init): Rename to haifa_insns_init. Update all uses. + (haifa_luid_for_non_insn): New static function. + (dep_weak): Move to sched-deps.c. Rename to ds_weak (). + (init_before_recovery): Use haifa_init_only_bb () instead of + add_block (). + (create_check_block_twin): Update to use new initialization functions. + (change_pattern): Rename to sched_change_pattern (). Make global. + Split haifa-specific functionality into ... + (haifa_change_pattern): New static function. + (speculate_insn): Rename to sched_speculate_insn (). Make global. + Split haifa-specific functionality into ... + (haifa_speculate_insn): New static function. + (sched_bbs_init, add_block, glat_init_1, glat_init, sched_bbs_finish): + Remove. + (debug_spec_status): Use get_dep_weak_1 () instead of get_dep_weak () + to prevent ICEs on malformed input. + (extend_bb, init_bb, extend_insn, init_insn, init_insns_in_bb): New + static functions for walking through scheduling region. + (sched_scan): New function for walking through scheduling region. + (sched_extend_bb, sched_init_bb): New static functions. + (sched_init_bbs, sched_finish_bbs): New functions to init / finalize + basic block information. + (attach_life_info1, attach_life_info): Move to the better place. + (sched_luids): New vector variable to replace uid_to_luid. + (sched_luid_extend): Rename to luids_extend_insn (). Update to use + sched_luids. + (sched_max_luid): New variable. + (sched_luid_init): Rename to luids_init_insn (). + (sched_init_luids, sched_finish_luids): New functions to + init / finalize logical uids. + (sched_luid_finish): Remove. + (sched_insn_extend, sched_insn_init, sched_insn_finish): Remove. + (sched_extend_target): New function. + (h_i_d): Make it a vector. + (extend_h_i_d, init_h_i_d, haifa_init_h_i_d, haifa_finish_h_i_d): + New functions to initialize / finalize haifa instruction data. + (haifa_init_insn): New static function. + (sched_bb_extend, sched_bb_init): Remove. + (haifa_init_only_bb): New static function. + (sched_data_update, sched_data_finish): Remove. + + * modulo-sched.c (sms_common_sched_info): Move initialization to ... + (setup_sched_infos): New static function. + (sms_schedule): Move initialization of common_sched_info, + sched_deps_info, current_sched_info to setup_sched_infos (). + + * sel-sched.c (reset_sched_cycles_p): New variable. + (old_ready_veclen): Rename to max_issue_size. + (substitute_rhs): Rename to substitute_reg_in_rhs (). Update to use + vinsns properly. + (replace_in_vinsn_using_bitmask): Rename to + replace_in_rtx_using_bitmask (). Update to use vinsns properly. + (un_substitute): Update to use vinsns properly. + (replace_src_with_reg): Rename to create_insn_rtx_with_rhs. Don't use + vinsns. + (replace_dest_with_reg_in_vinsn): Rename to create_insn_rtx_with_lhs. + Don't use vinsns. + (replace_dest_with_reg_in_rhs): Update. + (expr_dest_reg): New static function. + (rhs_dest_regno): Use expr_dest_reg (). + (mark_unavailable_hard_regs): Update. + (choose_best_reg): Add assertion. Move logic to ... + (choose_best_reg_1): New static function. + (choose_best_pseudo_reg): Add output parameter. + (find_best_reg_for_rhs): Change type of return value. Update. + (sel_speculation_p): New static variable. + (can_overcome_dep_p, create_speculation_check_insn_rtx): New static + functions. + (apply_spec_to_expr, speculate_expr, has_spec_dependence_p): Ditto. + (un_speculate): Ditto. + (moveup_rhs): Add support for data and control speculation. Update. + (moveup_set_rhs, equal_after_moveup_path_p, compute_av_set): Update. + (propagate_lv_set, compute_live, update_data_sets): Update. + (get_spec_check_type_for_insn): New static function. + (find_used_regs_1, find_used_regs, sel_rank_for_schedule): Update to + support speculation. + (fill_ready_list, find_best_rhs_and_reg_that_fits): Ditto. + (gen_insn_from_expr_after): New static function. + (generate_bookkeeping_insn, fill_insns): Update + (get_dest_reg_from_orig_ops): New static function. + (move_op): Update to support speculation. + (init_seqno): Add parameter. Update. + (sel_restore_other_notes): Move to sel-sched-ir.c. + (is_loop_preheader_p): Move to sel-sched-ir.c. Rename to + sel_is_loop_preheader_p (). + (sel_remove_loop_preheader): Move to sel-sched-ir.c. + (sel_region_init, sel_region_finish, sel_sched_region_1): Update. + (sel_global_init, sel_global_finish): Update. + (selective_scheduling_run): Add .dot dumping. + (handle_sel_sched): Use debugging parameter to choose scheduler. + + * sel-sched-ir.c (sel_bb_info): Change array to vector. + (sel_max_uid, lvs, lvs_size): Remove. + (s_i_r_d): New static vector variable. + (get_nop_from_pool): Rewrite. + (free_nop_pool): Update. + (vinsn_equal_insn_p): Remove. + (vinsn_separable_p): Rename to lhs_and_rhs_separable_p. Update. + (vinsn_init): Update. + (sel_insn_rtx_cost): New static function. + (sel_vinsn_cost, sel_gen_insn_from_rtx_after): New functions. + sel_gen_insn_from_expr_after): Ditto. + (rhs_equal_p): Rename to vinsn_correlate_as_rhses_p (). Update. + (rhs_init): Rename to init_expr (). Update. + (rhs_copy): Rename to copy_expr (). + (merge_expr_data): New function. + (rhs_merge): Rename to merge_expr (). + (rhs_equal_insn_p): Remove. + (clear_expr): New function. + (av_set_lookup_rhs): Rename to av_set_lookup (). Update. + (av_set_loopup_other_equiv_rhs): Update. + (av_set_remove_rhs_with_insn): Remove. + (av_set_add_vinsn): Rename to av_set_add (). Update. + (av_set_copy): Update. + (av_set_lookup_insn): Remove. + (av_set_add_insn): Remove. + (deps_init_id_start_insn, deps_init_id_finish_insn): Update. + (deps_init_id_start_lhs, deps_init_id_finish_lhs): Ditto. + (deps_init_id_start_rhs, deps_init_id_finish_rhs): Ditto. + (deps_init_id_reset_deps_to_insn): Rename to + deps_init_id_downgrade_to_use (). + (deps_init_id_note_reg_set, deps_init_id_note_reg_clobber): Ditto. + (deps_init_id_note_reg_use, deps_init_id_note_mem_dep): Ditto. + (sel_cfg_note_p): New static function. + (init_global_and_expr_for_insn): New static function. + (sel_init_global_and_expr): New function. + (finish_global_and_expr_insn_1, finish_global_and_expr_insn_1): New + static functions. + (sel_finish_global_and_expr): New function. + (sel_deps_*): Rename functions to has_dependence_*. Update. + (save_deps_info): Remove. + (setup_has_dependence_sched_deps_info): New static function. + (has_dependence_p): Update. + (tick_check_dep_with_dw, tick_check_p): Update. + (lhs_equals_reg_p): Rename to lhs_of_insn_equals_to_reg_p (). + (get_vinsn_type_for_insn): Remove. + (insn_valid_p): Rename to insn_rtx_valid. + (sel_insn_deffered_init): Remove. + (copy_insn_out_of_stream, copy_insn_and_insert_before): Remove. + (set_insn_init): New function. + (init_insn, init_simplejump, insn_init_move_lv_set_if_bb_header): New + static functions. + (sel_init_new_insns, sel_finish_new_insns): New functions. + (dfa_cost): Rename to vinsn_dfa_cost (). + (bb_header_p): Rename to sel_bb_header_p (). + (bb_empty_p): Rename to sel_bb_empty_p (). + (sel_insn_has_single_succ_p): New function. + (sel_add_or_remove_bb): Update. + (sel_create_basic_block_before, sel_merge_blocks): New function. + (create_insn_rtx_from_pattern_1): New static function. + (create_insn_rtx_from_pattern, create_vinsn_from_insn_rtx): New + functions. + (create_copy_of_insn_rtx, change_vinsn_in_expr): Ditto. + + * sel-sched-ir.h (struct _rhs): Rename to 'struct _expr'. + (struct _expr: sched_times, spec_done_ds, spec_to_check_ds): New + fields. + (struct vinsn_def: sched_cycle, sched_times, separable): Remove fields. + (struct vinsn_def: cost, may_trap_p): New fields. + (struct sel_insn_data: sched_cycle): New field. + (struct _sel_insn_rtx_data): New type. + + * sel-sched-dump.c (dump_insn_rtx, debug_insn_rtx): New functions. + (dump_vinsn, debug_vinsn, dump_expr, debug_expr): Ditto. + (dump_insn, debug_insn): Ditto. + (sel_dump_cfg_2): Update. + + * sel-sched-dump.h (enum _dump_insn_rtx, enum _dump_idata): New enums. + (enum _dump_vinsn, enum _dump_expr, enum _dump_insn): New enums. + + * emit-rtl.c (emit_insn_after_1): Call hook. + + * sched-deps.c (note_reg_use, note_reg_set, note_reg_clobber): Convert + macros to functions. + (note_mem_dep, note_dep): Ditto. + (sched_analyze_2): Generate control speculative dependencies. + (deps_analyze_insn): Update. + (d_i_d): Convert array to vector. + (deps_extend_d_i_d, deps_finish_d_i_d): New functions. + (sched_deps_local_init): Update. + (get_dep_weak): Move logic to ... + (get_dep_weak_1): New function. + (ds_merge): Move logic to ... + (ds_merge_1): New static function. + (ds_full_merge, ds_max_merge, ds_get_speculation_types): New functions. + (ds_get_max_dep_weak): New function. + + * sched-deps.h (sched-int.h): New include. + (struct deps_insn_data: depend): Move field to + 'struct haifa_deps_insn_data'. + + * target-def.h (TARGET_SCHED_GEN_CHECK): Rename to + TARGET_SCHED_GEN_SPEC_CHECK. + (TARGET_SCHED_GET_INSN_SPEC_DS, TARGET_SCHED_GET_INSN_CHECKED_DS): + New macros. + (TARGET_SCHED_SKIP_RTX_P): New macro. + + * sched-int.h (vecprim.h): New include. + (bb_vec_t, insn_vec_t): New typedefs. + (struct sched_scan_info_def): New type. + (sched_scan_info): Declare variable. + (sched_scan, sched_init_bbs, sched_finish_bbs, sched_init_luids): + Declare functions. + (sched_finish_luids, sched_extend_target, haifa_init_h_i_d): Ditto. + (haifa_finish_h_i_d, haifa_init_only_bb): Ditto. + (struct common_sched_info_def): Remove fields which_luid, remove_notes, + bb_extend, bb_init, bb_finish, insn_extend, insn_init. Add field + luid_for_new_insn. + (struct haifa_insn_data): Add fields reg_weight, priority_known. + + * Makefile.in (SCHED_INT_H, SCHED_DEPS_H): Update. + (fwprop.o): Fix dependencies. + (haifa-sched.o, sched-deps.o, sel-sched.o, sel-sched-ir.o): Update + dependencies. + + * sched-rgn.c (find_single_block_region, haifa_find_rgns): Update. + (extend_rgns, schedule_region, sched_rgn_finish): Ditto. + (sched_rgn_local_preinit, sched_rgn_local_init): Ditto. + (rgn_setup_common_sched_info, rgn_setup_sched_infos): New function. + + * sched-vis.c (dump_insn_slim_1): Update. + + * config/ia64/ia64.opt (msched-ldc): Rename to msched-spec-ldc. + (msched-control-ldc): Rename to msched-spec-control-ldc. + (msel-sched-data-spec, msel-sched-control-spec): New flags. + (msel-sched-dont-check-control-spec): New flag. + + * config/ia64/ia64.c: Rewrite speculation hooks. + + * config/ia64/ia64.md (speculable1, speculable2): New attributes. + + * params.def (PARAM_SELSCHED_MAX_SCHED_TIMES): New parameter. + (PARAM_ALLOW_START, PARAM_ALLOW_STOP, PARAM_ALLOW_P): Remove. + (PARAM_SEL1_START, PARAM_SEL1_STOP, PARAM_SEL1_P): New parameters. + (PARAM_SEL2_START, PARAM_SEL2_STOP, PARAM_SEL2_P): Ditto. + 2007-05-23 Alexander Monakov <amonakov@ispras.ru> * target.h (struct gcc_target): Add new parameter to adjust_cost_2 diff --git a/gcc/Makefile.in b/gcc/Makefile.in index 5b77ba29fbd..12854cd5f6b 100644 --- a/gcc/Makefile.in +++ b/gcc/Makefile.in @@ -746,9 +746,10 @@ EXPR_H = expr.h insn-config.h $(FUNCTION_H) $(RTL_H) $(FLAGS_H) $(TREE_H) $(MACH OPTABS_H = optabs.h insn-codes.h REGS_H = regs.h varray.h $(MACHMODE_H) $(OBSTACK_H) $(BASIC_BLOCK_H) $(FUNCTION_H) RESOURCE_H = resource.h hard-reg-set.h -SCHED_INT_H = sched-int.h $(INSN_ATTR_H) $(BASIC_BLOCK_H) sched-deps.h +SCHED_INT_H = sched-int.h $(INSN_ATTR_H) $(BASIC_BLOCK_H) vecprim.h +SCHED_DEPS_H = sched-deps.h $(SCHED_INT_H) SEL_SCHED_IR_H = sel-sched-ir.h $(INSN_ATTR_H) $(BASIC_BLOCK_H) $(RTL_H) \ - $(GGC_H) $(SCHED_INT_H) sched-rgn.h + $(GGC_H) $(SCHED_INT_H) sched-rgn.h SEL_SCHED_DUMP_H = sel-sched-dump.h $(SEL_SCHED_IR_H) INTEGRATE_H = integrate.h $(VARRAY_H) CFGLAYOUT_H = cfglayout.h $(BASIC_BLOCK_H) @@ -2481,7 +2482,7 @@ cse.o : cse.c $(CONFIG_H) $(SYSTEM_H) coretypes.h $(TM_H) $(RTL_H) $(REGS_H) \ except.h $(TARGET_H) $(PARAMS_H) rtlhooks-def.h tree-pass.h $(REAL_H) fwprop.o : fwprop.c $(CONFIG_H) $(SYSTEM_H) coretypes.h $(TM_H) $(RTL_H) \ toplev.h insn-config.h $(RECOG_H) $(FLAGS_H) $(OBSTACK_H) $(BASIC_BLOCK_H) \ - output.h $(DF_H) alloc-pool.h $(TIMEVAR_H) tree-pass.h + output.h $(DF_H) $(TARGET_H) alloc-pool.h $(TIMEVAR_H) tree-pass.h web.o : web.c $(CONFIG_H) $(SYSTEM_H) coretypes.h $(TM_H) $(RTL_H) \ hard-reg-set.h $(FLAGS_H) $(BASIC_BLOCK_H) $(FUNCTION_H) output.h toplev.h \ $(DF_H) $(OBSTACK_H) $(TIMEVAR_H) tree-pass.h @@ -2701,11 +2702,11 @@ modulo-sched.o : modulo-sched.c $(DDG_H) $(CONFIG_H) $(CONFIG_H) $(SYSTEM_H) \ haifa-sched.o : haifa-sched.c $(CONFIG_H) $(SYSTEM_H) coretypes.h $(TM_H) \ $(RTL_H) $(SCHED_INT_H) $(REGS_H) hard-reg-set.h $(FLAGS_H) insn-config.h \ $(FUNCTION_H) $(INSN_ATTR_H) toplev.h $(RECOG_H) except.h $(TM_P_H) \ - $(TARGET_H) output.h $(PARAMS_H) + $(TARGET_H) output.h $(PARAMS_H) vecprim.h sched-deps.o : sched-deps.c $(CONFIG_H) $(SYSTEM_H) coretypes.h $(TM_H) \ - $(RTL_H) sched-deps.h $(REGS_H) hard-reg-set.h $(FLAGS_H) insn-config.h \ - $(FUNCTION_H) $(INSN_ATTR_H) toplev.h $(RECOG_H) except.h cselib.h \ - $(PARAMS_H) $(TM_P_H) $(DF_H) + $(RTL_H) $(REGS_H) hard-reg-set.h $(FLAGS_H) insn-config.h \ + $(FUNCTION_H) $(INSN_ATTR_H) toplev.h $(RECOG_H) except.h $(SCHED_INT_H) \ + sched-deps.h $(PARAMS_H) cselib.h $(TM_P_H) $(DF_H) sched-rgn.o : sched-rgn.c $(CONFIG_H) $(SYSTEM_H) coretypes.h $(TM_H) \ $(RTL_H) $(SCHED_INT_H) $(REGS_H) hard-reg-set.h $(FLAGS_H) insn-config.h \ $(FUNCTION_H) $(INSN_ATTR_H) toplev.h $(RECOG_H) except.h $(PARAMS_H) \ @@ -2721,8 +2722,8 @@ sel-sched.o : sel-sched.c $(CONFIG_H) $(SYSTEM_H) coretypes.h $(TM_H) \ $(RTL_H) $(REGS_H) hard-reg-set.h $(FLAGS_H) insn-config.h \ $(FUNCTION_H) $(INSN_ATTR_H) toplev.h $(RECOG_H) except.h $(PARAMS_H) \ $(TM_P_H) $(TARGET_H) $(CFGLAYOUT_H) $(TIMEVAR_H) tree-pass.h sched-rgn.h \ - sched-deps.h $(GGC_H) $(TREE_H) $(LANGHOOKS_DEF_H) sel-sched.h \ - $(SEL_SCHED_IR_H) $(SEL_SCHED_DUMP_H) + $(SCHED_INT_H) sched-deps.h $(GGC_H) $(TREE_H) $(LANGHOOKS_DEF_H) \ + $(SEL_SCHED_IR_H) $(SEL_SCHED_DUMP_H) sel-sched.h sel-sched-dump.o : sel-sched-dump.c $(CONFIG_H) $(SYSTEM_H) coretypes.h $(TM_H) \ $(RTL_H) $(REGS_H) hard-reg-set.h $(FLAGS_H) insn-config.h \ $(FUNCTION_H) $(INSN_ATTR_H) toplev.h $(RECOG_H) except.h $(PARAMS_H) \ @@ -2732,8 +2733,7 @@ sel-sched-ir.o : sel-sched-ir.c $(CONFIG_H) $(SYSTEM_H) coretypes.h $(TM_H) \ $(RTL_H) $(REGS_H) hard-reg-set.h $(FLAGS_H) insn-config.h \ $(FUNCTION_H) $(INSN_ATTR_H) toplev.h $(RECOG_H) except.h $(PARAMS_H) \ $(TM_P_H) $(TARGET_H) $(CFGLAYOUT_H) $(TIMEVAR_H) tree-pass.h sched-rgn.h \ - sched-deps.h $(GGC_H) $(TREE_H) $(LANGHOOKS_DEF_H) $(SEL_SCHED_IR_H) \ - $(SEL_SCHED_DUMP_H) + sched-deps.h $(GGC_H) $(TREE_H) $(LANGHOOKS_DEF_H) $(SEL_SCHED_IR_H) final.o : final.c $(CONFIG_H) $(SYSTEM_H) coretypes.h $(TM_H) $(RTL_H) \ $(TREE_H) $(FLAGS_H) intl.h $(REGS_H) $(RECOG_H) conditions.h \ insn-config.h $(INSN_ATTR_H) $(FUNCTION_H) output.h hard-reg-set.h \ diff --git a/gcc/config/ia64/ia64.c b/gcc/config/ia64/ia64.c index dd07615d221..fc34efafbab 100644 --- a/gcc/config/ia64/ia64.c +++ b/gcc/config/ia64/ia64.c @@ -171,10 +171,12 @@ static void ia64_clear_sched_context (void *); static void ia64_free_sched_context (void *); static int ia64_mode_to_int (enum machine_mode); static void ia64_set_sched_flags (spec_info_t); +static ds_t ia64_get_insn_spec_ds (rtx); +static ds_t ia64_get_insn_checked_ds (rtx); +static bool ia64_skip_rtx_p (rtx); static int ia64_speculate_insn (rtx, ds_t, rtx *); -static rtx ia64_gen_spec_insn (rtx, ds_t, int, bool, bool); -static bool ia64_needs_block_p (rtx); -static rtx ia64_gen_check (rtx, rtx, bool); +static bool ia64_needs_block_p (int); +static rtx ia64_gen_spec_check (rtx, rtx, ds_t); static int ia64_spec_check_p (rtx); static int ia64_spec_check_src_p (rtx); static rtx gen_tls_get_addr (void); @@ -391,19 +393,28 @@ static const struct attribute_spec ia64_attribute_table[] = #undef TARGET_SCHED_SET_SCHED_FLAGS #define TARGET_SCHED_SET_SCHED_FLAGS ia64_set_sched_flags +#undef TARGET_SCHED_GET_INSN_SPEC_DS +#define TARGET_SCHED_GET_INSN_SPEC_DS ia64_get_insn_spec_ds + +#undef TARGET_SCHED_GET_INSN_CHECKED_DS +#define TARGET_SCHED_GET_INSN_CHECKED_DS ia64_get_insn_checked_ds + #undef TARGET_SCHED_SPECULATE_INSN #define TARGET_SCHED_SPECULATE_INSN ia64_speculate_insn #undef TARGET_SCHED_NEEDS_BLOCK_P #define TARGET_SCHED_NEEDS_BLOCK_P ia64_needs_block_p -#undef TARGET_SCHED_GEN_CHECK -#define TARGET_SCHED_GEN_CHECK ia64_gen_check +#undef TARGET_SCHED_GEN_SPEC_CHECK +#define TARGET_SCHED_GEN_SPEC_CHECK ia64_gen_spec_check #undef TARGET_SCHED_FIRST_CYCLE_MULTIPASS_DFA_LOOKAHEAD_GUARD_SPEC #define TARGET_SCHED_FIRST_CYCLE_MULTIPASS_DFA_LOOKAHEAD_GUARD_SPEC\ ia64_first_cycle_multipass_dfa_lookahead_guard_spec +#undef TARGET_SCHED_SKIP_RTX_P +#define TARGET_SCHED_SKIP_RTX_P ia64_skip_rtx_p + #undef TARGET_FUNCTION_OK_FOR_SIBCALL #define TARGET_FUNCTION_OK_FOR_SIBCALL ia64_function_ok_for_sibcall #undef TARGET_ARG_PARTIAL_BYTES @@ -5808,6 +5819,7 @@ rtx_needs_barrier (rtx x, struct reg_flags flags, int pred) case UNSPEC_FR_SQRT_RECIP_APPROX: case UNSPEC_LDA: case UNSPEC_LDS: + case UNSPEC_LDS_A: case UNSPEC_LDSA: case UNSPEC_CHKACLR: case UNSPEC_CHKS: @@ -6211,14 +6223,6 @@ static state_t prev_cycle_state = NULL; static char *stops_p = NULL; -/* The following array element values are ZERO for non-speculative - instructions and hold corresponding speculation check number for - speculative instructions. */ -static int *spec_check_no = NULL; - -/* Size of spec_check_no array. */ -static int max_uid = 0; - /* The following variable is used to set up the mentioned above array. */ static int stop_before_p = 0; @@ -6413,7 +6417,7 @@ ia64_sched_init_global (FILE *dump ATTRIBUTE_UNUSED, int sched_verbose ATTRIBUTE_UNUSED, int max_ready ATTRIBUTE_UNUSED) { - gcc_assert (!pending_data_specs); + gcc_assert (pending_data_specs == 0); } /* Scheduling pass is now finished. Free/reset static variable. */ @@ -6421,9 +6425,7 @@ static void ia64_sched_finish_global (FILE *dump ATTRIBUTE_UNUSED, int sched_verbose ATTRIBUTE_UNUSED) { - free (spec_check_no); - spec_check_no = 0; - max_uid = 0; + gcc_assert (pending_data_specs == 0); } /* We are about to being issuing insns for this clock cycle. @@ -6548,7 +6550,7 @@ ia64_variable_issue (FILE *dump ATTRIBUTE_UNUSED, rtx insn ATTRIBUTE_UNUSED, int can_issue_more ATTRIBUTE_UNUSED) { - if (sched_deps_info->generate_spec_deps) + if (sched_deps_info->generate_spec_deps && !SEL_SCHED_P) /* Modulo scheduling does not extend h_i_d when emitting new instructions. Don't use h_i_d, if we don't have to. */ { @@ -6727,15 +6729,6 @@ ia64_dfa_new_cycle (FILE *dump, int verbose, rtx insn, int last_clock, static void ia64_h_i_d_extended (void) { - if (sched_deps_info->generate_spec_deps) - { - int new_max_uid = get_max_uid () + 1; - - spec_check_no = xrecalloc (spec_check_no, new_max_uid, - max_uid, sizeof (*spec_check_no)); - max_uid = new_max_uid; - } - if (stops_p != NULL) { int new_clocks_length = get_max_uid () + 1; @@ -6847,83 +6840,55 @@ ia64_free_sched_context (void *_sc) free (_sc); } -/* Constants that help mapping 'enum machine_mode' to int. */ -enum SPEC_MODES - { - SPEC_MODE_INVALID = -1, - SPEC_MODE_FIRST = 0, - SPEC_MODE_FOR_EXTEND_FIRST = 1, - SPEC_MODE_FOR_EXTEND_LAST = 3, - SPEC_MODE_LAST = 8 - }; - -/* Return index of the MODE. */ -static int -ia64_mode_to_int (enum machine_mode mode) -{ - switch (mode) - { - case BImode: return 0; /* SPEC_MODE_FIRST */ - case QImode: return 1; /* SPEC_MODE_FOR_EXTEND_FIRST */ - case HImode: return 2; - case SImode: return 3; /* SPEC_MODE_FOR_EXTEND_LAST */ - case DImode: return 4; - case SFmode: return 5; - case DFmode: return 6; - case XFmode: return 7; - case TImode: - /* ??? This mode needs testing. Bypasses for ldfp8 instruction are not - mentioned in itanium[12].md. Predicate fp_register_operand also - needs to be defined. Bottom line: better disable for now. */ - return SPEC_MODE_INVALID; - default: return SPEC_MODE_INVALID; - } -} - /* Provide information about speculation capabilities. */ static void ia64_set_sched_flags (spec_info_t spec_info) { if (common_sched_info->sched_pass_id == SCHED_RGN_PASS - || common_sched_info->sched_pass_id == SCHED_EBB_PASS) + || common_sched_info->sched_pass_id == SCHED_EBB_PASS + || SEL_SCHED_P) { int mask = 0; - if ((mflag_sched_br_data_spec && !reload_completed && optimize > 0) - || (mflag_sched_ar_data_spec && reload_completed)) + if ((!SEL_SCHED_P + && ((mflag_sched_br_data_spec && !reload_completed && optimize > 0) + || (mflag_sched_ar_data_spec && reload_completed))) + || (SEL_SCHED_P && mflag_sel_sched_data_spec)) { mask |= BEGIN_DATA; - - if ((mflag_sched_br_in_data_spec && !reload_completed) - || (mflag_sched_ar_in_data_spec && reload_completed)) + + if (!SEL_SCHED_P + && ((mflag_sched_br_in_data_spec && !reload_completed) + || (mflag_sched_ar_in_data_spec && reload_completed))) mask |= BE_IN_DATA; } - if (mflag_sched_control_spec) + if ((!SEL_SCHED_P && mflag_sched_control_spec) + || (SEL_SCHED_P && mflag_sel_sched_control_spec)) { mask |= BEGIN_CONTROL; - if (mflag_sched_in_control_spec) + if (!SEL_SCHED_P && mflag_sched_in_control_spec) mask |= BE_IN_CONTROL; } - gcc_assert (common_sched_info->use_glat); + spec_info->mask = mask; if (mask) { - common_sched_info->detach_life_info = 1; - sched_deps_info->use_deps_list = 1; - sched_deps_info->generate_spec_deps = 1; - - spec_info->mask = mask; spec_info->flags = 0; if ((mask & DATA_SPEC) && mflag_sched_prefer_non_data_spec_insns) spec_info->flags |= PREFER_NON_DATA_SPEC; - if ((mask & CONTROL_SPEC) - && mflag_sched_prefer_non_control_spec_insns) - spec_info->flags |= PREFER_NON_CONTROL_SPEC; + if (mask & CONTROL_SPEC) + { + if (mflag_sched_prefer_non_control_spec_insns) + spec_info->flags |= PREFER_NON_CONTROL_SPEC; + + if (SEL_SCHED_P && mflag_sel_sched_dont_check_control_spec) + spec_info->flags |= SEL_SCHED_SPEC_DONT_CHECK_CONTROL; + } if (mflag_sched_spec_verbose) { @@ -6941,120 +6906,29 @@ ia64_set_sched_flags (spec_info_t spec_info) } } -/* Implement targetm.sched.speculate_insn hook. - Check if the INSN can be TS speculative. - If 'no' - return -1. - If 'yes' - generate speculative pattern in the NEW_PAT and return 1. - If current pattern of the INSN already provides TS speculation, return 0. */ -static int -ia64_speculate_insn (rtx insn, ds_t ts, rtx *new_pat) -{ - rtx pat, reg, mem, mem_reg; - int mode_no, gen_p = 1; - bool extend_p; - - gcc_assert (!(ts & ~BEGIN_SPEC) && ts); - - pat = PATTERN (insn); - - if (GET_CODE (pat) == COND_EXEC) - pat = COND_EXEC_CODE (pat); - - if (GET_CODE (pat) != SET) - return -1; - reg = SET_DEST (pat); - if (!REG_P (reg)) - return -1; - - mem = SET_SRC (pat); - if (GET_CODE (mem) == ZERO_EXTEND) - { - mem = XEXP (mem, 0); - extend_p = true; - } - else - extend_p = false; - - if (GET_CODE (mem) == UNSPEC) - { - int code; - - code = XINT (mem, 1); - if (code != UNSPEC_LDA && code != UNSPEC_LDS && code != UNSPEC_LDSA) - return -1; - - if ((code == UNSPEC_LDA && !(ts & BEGIN_CONTROL)) - || (code == UNSPEC_LDS && !(ts & BEGIN_DATA)) - || code == UNSPEC_LDSA) - gen_p = 0; - - mem = XVECEXP (mem, 0, 0); - gcc_assert (MEM_P (mem)); - } - if (!MEM_P (mem)) - return -1; - mem_reg = XEXP (mem, 0); - if (!REG_P (mem_reg)) - return -1; - - /* We should use MEM's mode since REG's mode in presence of ZERO_EXTEND - will always be DImode. */ - mode_no = ia64_mode_to_int (GET_MODE (mem)); - - if (mode_no == SPEC_MODE_INVALID - || (extend_p - && !(SPEC_MODE_FOR_EXTEND_FIRST <= mode_no - && mode_no <= SPEC_MODE_FOR_EXTEND_LAST))) - return -1; - - extract_insn_cached (insn); - gcc_assert (reg == recog_data.operand[0] && mem == recog_data.operand[1]); - *new_pat = ia64_gen_spec_insn (insn, ts, mode_no, gen_p != 0, extend_p); - - return gen_p; -} - -enum - { - /* Offset to reach ZERO_EXTEND patterns. */ - SPEC_GEN_EXTEND_OFFSET = SPEC_MODE_LAST - SPEC_MODE_FOR_EXTEND_FIRST + 1, - /* Number of patterns for each speculation mode. */ - SPEC_N = (SPEC_MODE_LAST - + SPEC_MODE_FOR_EXTEND_LAST - SPEC_MODE_FOR_EXTEND_FIRST + 2) - }; +typedef rtx (* gen_func_t) (rtx, rtx); -enum SPEC_GEN_LD_MAP - { - /* Offset to ld.a patterns. */ - SPEC_GEN_A = 0 * SPEC_N, - /* Offset to ld.s patterns. */ - SPEC_GEN_S = 1 * SPEC_N, - /* Offset to ld.sa patterns. */ - SPEC_GEN_SA = 2 * SPEC_N, - /* Offset to ld.sa patterns. For this patterns corresponding ld.c will - mutate to chk.s. */ - SPEC_GEN_SA_FOR_S = 3 * SPEC_N - }; - -/* These offsets are used to get (4 * SPEC_N). */ -enum SPEC_GEN_CHECK_OFFSET - { - SPEC_GEN_CHKA_FOR_A_OFFSET = 4 * SPEC_N - SPEC_GEN_A, - SPEC_GEN_CHKA_FOR_SA_OFFSET = 4 * SPEC_N - SPEC_GEN_SA - }; - -/* If GEN_P is true, calculate the index of needed speculation check and return - speculative pattern for INSN with speculative mode TS, machine mode - MODE_NO and with ZERO_EXTEND (if EXTEND_P is true). - If GEN_P is false, just calculate the index of needed speculation check. */ -static rtx -ia64_gen_spec_insn (rtx insn, ds_t ts, int mode_no, bool gen_p, bool extend_p) +/* Return a function that will generate a load of mode MODE_NO + with speculation types TS. */ +static gen_func_t +get_spec_load_gen_function (ds_t ts, int mode_no) { - rtx pat, new_pat; - int load_no; - int shift = 0; + static gen_func_t gen_ld_[] = { + gen_movbi, + gen_movqi_internal, + gen_movhi_internal, + gen_movsi_internal, + gen_movdi_internal, + gen_movsf_internal, + gen_movdf_internal, + gen_movxf_internal, + gen_movti_internal, + gen_zero_extendqidi2, + gen_zero_extendhidi2, + gen_zero_extendsidi2, + }; - static rtx (* const gen_load[]) (rtx, rtx) = { + static gen_func_t gen_ld_a[] = { gen_movbi_advanced, gen_movqi_advanced, gen_movhi_advanced, @@ -7067,7 +6941,8 @@ ia64_gen_spec_insn (rtx insn, ds_t ts, int mode_no, bool gen_p, bool extend_p) gen_zero_extendqidi2_advanced, gen_zero_extendhidi2_advanced, gen_zero_extendsidi2_advanced, - + }; + static gen_func_t gen_ld_s[] = { gen_movbi_speculative, gen_movqi_speculative, gen_movhi_speculative, @@ -7080,7 +6955,8 @@ ia64_gen_spec_insn (rtx insn, ds_t ts, int mode_no, bool gen_p, bool extend_p) gen_zero_extendqidi2_speculative, gen_zero_extendhidi2_speculative, gen_zero_extendsidi2_speculative, - + }; + static gen_func_t gen_ld_sa[] = { gen_movbi_speculative_advanced, gen_movqi_speculative_advanced, gen_movhi_speculative_advanced, @@ -7093,140 +6969,359 @@ ia64_gen_spec_insn (rtx insn, ds_t ts, int mode_no, bool gen_p, bool extend_p) gen_zero_extendqidi2_speculative_advanced, gen_zero_extendhidi2_speculative_advanced, gen_zero_extendsidi2_speculative_advanced, - - gen_movbi_speculative_advanced, - gen_movqi_speculative_advanced, - gen_movhi_speculative_advanced, - gen_movsi_speculative_advanced, - gen_movdi_speculative_advanced, - gen_movsf_speculative_advanced, - gen_movdf_speculative_advanced, - gen_movxf_speculative_advanced, - gen_movti_speculative_advanced, - gen_zero_extendqidi2_speculative_advanced, - gen_zero_extendhidi2_speculative_advanced, - gen_zero_extendsidi2_speculative_advanced + }; + static gen_func_t gen_ld_s_a[] = { + gen_movbi_speculative_a, + gen_movqi_speculative_a, + gen_movhi_speculative_a, + gen_movsi_speculative_a, + gen_movdi_speculative_a, + gen_movsf_speculative_a, + gen_movdf_speculative_a, + gen_movxf_speculative_a, + gen_movti_speculative_a, + gen_zero_extendqidi2_speculative_a, + gen_zero_extendhidi2_speculative_a, + gen_zero_extendsidi2_speculative_a, }; - load_no = extend_p ? mode_no + SPEC_GEN_EXTEND_OFFSET : mode_no; + gen_func_t *gen_ld; if (ts & BEGIN_DATA) { - /* We don't need recovery because even if this is ld.sa - ALAT entry will be allocated only if NAT bit is set to zero. - So it is enough to use ld.c here. */ - if (ts & BEGIN_CONTROL) - { - load_no += SPEC_GEN_SA; - - if (!mflag_sched_ldc) - shift = SPEC_GEN_CHKA_FOR_SA_OFFSET; - } + gen_ld = gen_ld_sa; else - { - load_no += SPEC_GEN_A; - - if (!mflag_sched_ldc) - shift = SPEC_GEN_CHKA_FOR_A_OFFSET; - } + gen_ld = gen_ld_a; } else if (ts & BEGIN_CONTROL) { - /* ld.sa can be used instead of ld.s to avoid basic block splitting. */ - if (!mflag_control_ldc) - load_no += SPEC_GEN_S; + if ((spec_info->flags & SEL_SCHED_SPEC_DONT_CHECK_CONTROL) + || ia64_needs_block_p (ts)) + gen_ld = gen_ld_s; else + gen_ld = gen_ld_s_a; + } + else if (ts == 0) + gen_ld = gen_ld_; + else + gcc_unreachable (); + + return gen_ld[mode_no]; +} + +/* Constants that help mapping 'enum machine_mode' to int. */ +enum SPEC_MODES + { + SPEC_MODE_INVALID = -1, + SPEC_MODE_FIRST = 0, + SPEC_MODE_FOR_EXTEND_FIRST = 1, + SPEC_MODE_FOR_EXTEND_LAST = 3, + SPEC_MODE_LAST = 8 + }; + +enum + { + /* Offset to reach ZERO_EXTEND patterns. */ + SPEC_GEN_EXTEND_OFFSET = SPEC_MODE_LAST - SPEC_MODE_FOR_EXTEND_FIRST + 1 + }; + +/* Return index of the MODE. */ +static int +ia64_mode_to_int (enum machine_mode mode) +{ + switch (mode) + { + case BImode: return 0; /* SPEC_MODE_FIRST */ + case QImode: return 1; /* SPEC_MODE_FOR_EXTEND_FIRST */ + case HImode: return 2; + case SImode: return 3; /* SPEC_MODE_FOR_EXTEND_LAST */ + case DImode: return 4; + case SFmode: return 5; + case DFmode: return 6; + case XFmode: return 7; + case TImode: + /* ??? This mode needs testing. Bypasses for ldfp8 instruction are not + mentioned in itanium[12].md. Predicate fp_register_operand also + needs to be defined. Bottom line: better disable for now. */ + return SPEC_MODE_INVALID; + default: return SPEC_MODE_INVALID; + } +} + +/* If INSN is an appropriate load return its mode. + Return -1 otherwise. */ +static int +get_mode_no_for_insn (rtx insn) +{ + rtx reg, mem, mode_rtx; + int mode_no; + bool extend_p; + + extract_insn_cached (insn); + + /* We use WHICH_ALTERNATIVE only after reload. This will + guarantee that reload won't touch a speculative insn. */ + + if (recog_data.n_operands != 2) + return -1; + + reg = recog_data.operand[0]; + mem = recog_data.operand[1]; + + /* We should use MEM's mode since REG's mode in presence of + ZERO_EXTEND will always be DImode. */ + if (get_attr_speculable1 (insn) == SPECULABLE1_YES) + /* Process non-speculative ld. */ + { + if (!reload_completed) + { + if (!REG_P (reg)) + return -1; + + if (!MEM_P (mem)) + return -1; + + { + rtx mem_reg = XEXP (mem, 0); + + if (!REG_P (mem_reg)) + return -1; + } + + mode_rtx = mem; + } + else if (get_attr_speculable2 (insn) == SPECULABLE2_YES) { - gcc_assert (mflag_sched_ldc); - load_no += SPEC_GEN_SA_FOR_S; + gcc_assert (REG_P (reg) && MEM_P (mem)); + mode_rtx = mem; } + else + return -1; + } + else if (get_attr_data_speculative (insn) == DATA_SPECULATIVE_YES + || get_attr_control_speculative (insn) == CONTROL_SPECULATIVE_YES + || get_attr_check_load (insn) == CHECK_LOAD_YES) + /* Process speculative ld or ld.c. */ + { + gcc_assert (REG_P (reg) && MEM_P (mem)); + mode_rtx = mem; } else - gcc_unreachable (); + { + enum attr_itanium_class class = get_attr_itanium_class (insn); - /* Set the desired check index. We add '1', because zero element in this - array means, that instruction with such uid is non-speculative. */ - spec_check_no[INSN_UID (insn)] = load_no + shift + 1; + if (class == ITANIUM_CLASS_CHK_A || class == ITANIUM_CLASS_CHK_S_I + || class == ITANIUM_CLASS_CHK_S_F) + /* Process chk. */ + mode_rtx = reg; + else + return -1; + } - if (!gen_p) - return 0; + mode_no = ia64_mode_to_int (GET_MODE (mode_rtx)); + + if (mode_no == SPEC_MODE_INVALID) + return -1; - new_pat = gen_load[load_no] (copy_rtx (recog_data.operand[0]), - copy_rtx (recog_data.operand[1])); + extend_p = (GET_MODE (reg) != GET_MODE (mode_rtx)); + + if (extend_p) + { + if (!(SPEC_MODE_FOR_EXTEND_FIRST <= mode_no + && mode_no <= SPEC_MODE_FOR_EXTEND_LAST)) + return -1; + + mode_no += SPEC_GEN_EXTEND_OFFSET; + } + + return mode_no; +} + +/* If X is an unspec part of a speculative load, return its code. + Return -1 otherwise. */ +static int +get_spec_unspec_code (rtx x) +{ + if (GET_CODE (x) != UNSPEC) + return -1; + + { + int code; + + code = XINT (x, 1); + + switch (code) + { + case UNSPEC_LDA: + case UNSPEC_LDS: + case UNSPEC_LDS_A: + case UNSPEC_LDSA: + return code; + + default: + return -1; + } + } +} + +/* Implement skip_rtx_p hook. */ +static bool +ia64_skip_rtx_p (rtx x) +{ + return get_spec_unspec_code (x) != -1; +} + +/* If INSN is a speculative load, return its UNSPEC code. + Return -1 otherwise. */ +static int +get_insn_spec_code (rtx insn) +{ + rtx pat, reg, mem; pat = PATTERN (insn); + if (GET_CODE (pat) == COND_EXEC) - new_pat = gen_rtx_COND_EXEC (VOIDmode, copy_rtx - (COND_EXEC_TEST (pat)), new_pat); + pat = COND_EXEC_CODE (pat); - return new_pat; + if (GET_CODE (pat) != SET) + return -1; + + reg = SET_DEST (pat); + if (!REG_P (reg)) + return -1; + + mem = SET_SRC (pat); + if (GET_CODE (mem) == ZERO_EXTEND) + mem = XEXP (mem, 0); + + return get_spec_unspec_code (mem); } -/* Offset to branchy checks. */ -enum { SPEC_GEN_CHECK_MUTATION_OFFSET = 5 * SPEC_N }; +/* If INSN is a speculative load, return a ds with the speculation types. + Otherwise [if INSN is a normal instruction] return 0. */ +static ds_t +ia64_get_insn_spec_ds (rtx insn) +{ + int code = get_insn_spec_code (insn); + + switch (code) + { + case UNSPEC_LDA: + return BEGIN_DATA; -/* Return nonzero, if INSN needs branchy recovery check. */ -static bool -ia64_needs_block_p (rtx insn) + case UNSPEC_LDS: + case UNSPEC_LDS_A: + return BEGIN_CONTROL; + + case UNSPEC_LDSA: + return BEGIN_DATA | BEGIN_CONTROL; + + default: + return 0; + } +} + +/* If INSN is a speculative load return a ds with the speculation types that + will be checked. + Otherwise [if INSN is a normal instruction] return 0. */ +static ds_t +ia64_get_insn_checked_ds (rtx insn) { - int check_no; + int code = get_insn_spec_code (insn); - check_no = spec_check_no[INSN_UID(insn)] - 1; - gcc_assert (0 <= check_no && check_no < SPEC_GEN_CHECK_MUTATION_OFFSET); + switch (code) + { + case UNSPEC_LDA: + return BEGIN_DATA | BEGIN_CONTROL; - return ((SPEC_GEN_S <= check_no && check_no < SPEC_GEN_S + SPEC_N) - || (4 * SPEC_N <= check_no && check_no < 4 * SPEC_N + SPEC_N)); + case UNSPEC_LDS: + return BEGIN_CONTROL; + + case UNSPEC_LDS_A: + case UNSPEC_LDSA: + return BEGIN_DATA | BEGIN_CONTROL; + + default: + return 0; + } } -/* Generate (or regenerate, if (MUTATE_P)) recovery check for INSN. - If (LABEL != 0 || MUTATE_P), generate branchy recovery check. - Otherwise, generate a simple check. */ +/* If GEN_P is true, calculate the index of needed speculation check and return + speculative pattern for INSN with speculative mode TS, machine mode + MODE_NO and with ZERO_EXTEND (if EXTEND_P is true). + If GEN_P is false, just calculate the index of needed speculation check. */ static rtx -ia64_gen_check (rtx insn, rtx label, bool mutate_p) +ia64_gen_spec_load (rtx insn, ds_t ts, int mode_no) { - rtx op1, pat, check_pat; + rtx pat, new_pat; + gen_func_t gen_load; - static rtx (* const gen_check[]) (rtx, rtx) = { - gen_movbi_clr, - gen_movqi_clr, - gen_movhi_clr, - gen_movsi_clr, - gen_movdi_clr, - gen_movsf_clr, - gen_movdf_clr, - gen_movxf_clr, - gen_movti_clr, - gen_zero_extendqidi2_clr, - gen_zero_extendhidi2_clr, - gen_zero_extendsidi2_clr, + gen_load = get_spec_load_gen_function (ts, mode_no); - gen_speculation_check_bi, - gen_speculation_check_qi, - gen_speculation_check_hi, - gen_speculation_check_si, - gen_speculation_check_di, - gen_speculation_check_sf, - gen_speculation_check_df, - gen_speculation_check_xf, - gen_speculation_check_ti, - gen_speculation_check_di, - gen_speculation_check_di, - gen_speculation_check_di, + new_pat = gen_load (copy_rtx (recog_data.operand[0]), + copy_rtx (recog_data.operand[1])); - gen_movbi_clr, - gen_movqi_clr, - gen_movhi_clr, - gen_movsi_clr, - gen_movdi_clr, - gen_movsf_clr, - gen_movdf_clr, - gen_movxf_clr, - gen_movti_clr, - gen_zero_extendqidi2_clr, - gen_zero_extendhidi2_clr, - gen_zero_extendsidi2_clr, + pat = PATTERN (insn); + if (GET_CODE (pat) == COND_EXEC) + new_pat = gen_rtx_COND_EXEC (VOIDmode, copy_rtx (COND_EXEC_TEST (pat)), + new_pat); + + return new_pat; +} + +/* Implement targetm.sched.speculate_insn hook. + Check if the INSN can be TS speculative. + If 'no' - return -1. + If 'yes' - generate speculative pattern in the NEW_PAT and return 1. + If current pattern of the INSN already provides TS speculation, + return 0. */ +static int +ia64_speculate_insn (rtx insn, ds_t ts, rtx *new_pat) +{ + int mode_no; + int res; + + gcc_assert (!(ts & ~BEGIN_SPEC)); + + if (ia64_spec_check_p (insn)) + return -1; + + mode_no = get_mode_no_for_insn (insn); + if (mode_no != SPEC_MODE_INVALID) + { + if (0 && SEL_SCHED_P && (ts & BEGIN_DATA) + && mode_no >= 5 && mode_no <= 7) + /* !!! We can't data speculate float point loads because we might + get a trap-looking insn from the trap-free one. This happens + because of the UNSPEC:?F part in the load, that is a floating + point operation that might trap. */ + return -1; + + if (ia64_get_insn_spec_ds (insn) == ts) + res = 0; + else + { + res = 1; + *new_pat = ia64_gen_spec_load (insn, ts, mode_no); + } + } + else + res = -1; + + return res; +} + +/* Return a function that will generate a check for speculation TS with mode + MODE_NO. + If simple check is needed, pass true for SIMPLE_CHECK_P. + If clearing check is needed, pass true for CLEARING_CHECK_P. */ +static gen_func_t +get_spec_check_gen_function (ds_t ts, int mode_no, + bool simple_check_p, bool clearing_check_p) +{ + static gen_func_t gen_ld_c_clr[] = { gen_movbi_clr, gen_movqi_clr, gen_movhi_clr, @@ -7239,21 +7334,22 @@ ia64_gen_check (rtx insn, rtx label, bool mutate_p) gen_zero_extendqidi2_clr, gen_zero_extendhidi2_clr, gen_zero_extendsidi2_clr, - - gen_advanced_load_check_clr_bi, - gen_advanced_load_check_clr_qi, - gen_advanced_load_check_clr_hi, - gen_advanced_load_check_clr_si, - gen_advanced_load_check_clr_di, - gen_advanced_load_check_clr_sf, - gen_advanced_load_check_clr_df, - gen_advanced_load_check_clr_xf, - gen_advanced_load_check_clr_ti, - gen_advanced_load_check_clr_di, - gen_advanced_load_check_clr_di, - gen_advanced_load_check_clr_di, - - /* Following checks are generated during mutation. */ + }; + static gen_func_t gen_ld_c_nc[] = { + gen_movbi_nc, + gen_movqi_nc, + gen_movhi_nc, + gen_movsi_nc, + gen_movdi_nc, + gen_movsf_nc, + gen_movdf_nc, + gen_movxf_nc, + gen_movti_nc, + gen_zero_extendqidi2_nc, + gen_zero_extendhidi2_nc, + gen_zero_extendsidi2_nc, + }; + static gen_func_t gen_chk_a_clr[] = { gen_advanced_load_check_clr_bi, gen_advanced_load_check_clr_qi, gen_advanced_load_check_clr_hi, @@ -7266,22 +7362,22 @@ ia64_gen_check (rtx insn, rtx label, bool mutate_p) gen_advanced_load_check_clr_di, gen_advanced_load_check_clr_di, gen_advanced_load_check_clr_di, - - 0,0,0,0,0,0,0,0,0,0,0,0, - - gen_advanced_load_check_clr_bi, - gen_advanced_load_check_clr_qi, - gen_advanced_load_check_clr_hi, - gen_advanced_load_check_clr_si, - gen_advanced_load_check_clr_di, - gen_advanced_load_check_clr_sf, - gen_advanced_load_check_clr_df, - gen_advanced_load_check_clr_xf, - gen_advanced_load_check_clr_ti, - gen_advanced_load_check_clr_di, - gen_advanced_load_check_clr_di, - gen_advanced_load_check_clr_di, - + }; + static gen_func_t gen_chk_a_nc[] = { + gen_advanced_load_check_nc_bi, + gen_advanced_load_check_nc_qi, + gen_advanced_load_check_nc_hi, + gen_advanced_load_check_nc_si, + gen_advanced_load_check_nc_di, + gen_advanced_load_check_nc_sf, + gen_advanced_load_check_nc_df, + gen_advanced_load_check_nc_xf, + gen_advanced_load_check_nc_ti, + gen_advanced_load_check_nc_di, + gen_advanced_load_check_nc_di, + gen_advanced_load_check_nc_di, + }; + static gen_func_t gen_chk_s[] = { gen_speculation_check_bi, gen_speculation_check_qi, gen_speculation_check_hi, @@ -7293,48 +7389,97 @@ ia64_gen_check (rtx insn, rtx label, bool mutate_p) gen_speculation_check_ti, gen_speculation_check_di, gen_speculation_check_di, - gen_speculation_check_di + gen_speculation_check_di, }; - extract_insn_cached (insn); + gen_func_t *gen_check; - if (label) - { - gcc_assert (mutate_p || ia64_needs_block_p (insn)); - op1 = label; - } - else + if (ts & BEGIN_DATA) { - gcc_assert (!mutate_p && !ia64_needs_block_p (insn)); - op1 = copy_rtx (recog_data.operand[1]); + /* We don't need recovery because even if this is ld.sa + ALAT entry will be allocated only if NAT bit is set to zero. + So it is enough to use ld.c here. */ + + if (simple_check_p) + { + gcc_assert (mflag_sched_spec_ldc); + + if (clearing_check_p) + gen_check = gen_ld_c_clr; + else + gen_check = gen_ld_c_nc; + } + else + { + if (clearing_check_p) + gen_check = gen_chk_a_clr; + else + gen_check = gen_chk_a_nc; + } } - - if (mutate_p) - /* INSN is ld.c. - Find the speculation check number by searching for original - speculative load in the RESOLVED_DEPS list of INSN. - As long as patterns are unique for each instruction, this can be - accomplished by matching ORIG_PAT fields. */ + else if (ts & BEGIN_CONTROL) { - rtx link; - int check_no = 0; - rtx orig_pat = ORIG_PAT (insn); + if (simple_check_p) + /* We might want to use ld.sa -> ld.c instead of + ld.s -> chk.s. */ + { + gcc_assert (!ia64_needs_block_p (ts)); - for (link = RESOLVED_DEPS (insn); link; link = XEXP (link, 1)) + if (clearing_check_p) + gen_check = gen_ld_c_clr; + else + gen_check = gen_ld_c_nc; + } + else { - rtx x = XEXP (link, 0); + gcc_assert (clearing_check_p); - if (ORIG_PAT (x) == orig_pat) - check_no = spec_check_no[INSN_UID (x)]; + gen_check = gen_chk_s; } - gcc_assert (check_no); + } + else + gcc_unreachable (); + + return gen_check[mode_no]; +} + +/* Return nonzero, if INSN needs branchy recovery check. */ +static bool +ia64_needs_block_p (ds_t ts) +{ + if (ts & BEGIN_DATA) + return !mflag_sched_spec_ldc; + + gcc_assert ((ts & BEGIN_CONTROL) != 0); + + return (!((mflag_sched_spec_control_ldc && mflag_sched_spec_ldc) + || SEL_SCHED_P)); +} + +/* Generate (or regenerate, if (MUTATE_P)) recovery check for INSN. + If (LABEL != 0 || MUTATE_P), generate branchy recovery check. + Otherwise, generate a simple check. */ +static rtx +ia64_gen_spec_check (rtx insn, rtx label, ds_t ds) +{ + rtx op1, pat, check_pat; + gen_func_t gen_check; + int mode_no; + + mode_no = get_mode_no_for_insn (insn); - spec_check_no[INSN_UID (insn)] = (check_no - + SPEC_GEN_CHECK_MUTATION_OFFSET); + if (label) + op1 = label; + else + { + gcc_assert (!ia64_needs_block_p (ds)); + op1 = copy_rtx (recog_data.operand[1]); } + + gen_check = get_spec_check_gen_function (ds, mode_no, label == NULL_RTX, + true); - check_pat = (gen_check[spec_check_no[INSN_UID (insn)] - 1] - (copy_rtx (recog_data.operand[0]), op1)); + check_pat = gen_check (copy_rtx (recog_data.operand[0]), op1); pat = PATTERN (insn); if (GET_CODE (pat) == COND_EXEC) @@ -7375,9 +7520,11 @@ ia64_spec_check_src_p (rtx src) code = XINT (t, 1); - if (code == UNSPEC_CHKACLR - || code == UNSPEC_CHKS - || code == UNSPEC_LDCCLR) + if (code == UNSPEC_LDCCLR + || code == UNSPEC_LDCNC + || code == UNSPEC_CHKACLR + || code == UNSPEC_CHKANC + || code == UNSPEC_CHKS) { gcc_assert (code != 0); return code; @@ -8558,7 +8705,8 @@ ia64_ld_address_bypass_p (rtx producer, rtx consumer) { int c = XINT (mem, 1); - gcc_assert (c == UNSPEC_LDA || c == UNSPEC_LDS || c == UNSPEC_LDSA); + gcc_assert (c == UNSPEC_LDA || c == UNSPEC_LDS || c == UNSPEC_LDS_A + || c == UNSPEC_LDSA); mem = XVECEXP (mem, 0, 0); } @@ -8643,6 +8791,8 @@ emit_predicate_relation_info (void) } } +static int sel2_run = 0; + /* Perform machine dependent operations on the rtl chain INSNS. */ static void @@ -8735,10 +8885,28 @@ ia64_reorg (void) _1mfb_ = get_cpu_unit_code ("1b_1mfb."); _1mlx_ = get_cpu_unit_code ("1b_1mlx."); } - if (flag_selective_scheduling2) - selective_scheduling_run (); - else - schedule_ebbs (); + { + int now; + int start; + int stop; + bool do_p; + + now = ++sel2_run; + start = PARAM_VALUE (PARAM_SEL2_START); + stop = PARAM_VALUE (PARAM_SEL2_STOP); + do_p = (PARAM_VALUE (PARAM_SEL2_P) == 1); + + if (do_p) + do_p = (start <= now) && (now <= stop); + else + do_p = (start > now) || (now > stop); + + if (flag_selective_scheduling2 && do_p) + selective_scheduling_run (); + else + schedule_ebbs (); + } + finish_bundle_states (); if (ia64_tune == PROCESSOR_ITANIUM) { diff --git a/gcc/config/ia64/ia64.md b/gcc/config/ia64/ia64.md index f25ad76be16..3dd0ef6b0e2 100644 --- a/gcc/config/ia64/ia64.md +++ b/gcc/config/ia64/ia64.md @@ -83,10 +83,13 @@ (UNSPEC_VECT_EXTR 31) (UNSPEC_LDA 40) (UNSPEC_LDS 41) - (UNSPEC_LDSA 42) - (UNSPEC_LDCCLR 43) - (UNSPEC_CHKACLR 45) - (UNSPEC_CHKS 47) + (UNSPEC_LDS_A 42) + (UNSPEC_LDSA 43) + (UNSPEC_LDCCLR 44) + (UNSPEC_LDCNC 45) + (UNSPEC_CHKACLR 46) + (UNSPEC_CHKANC 47) + (UNSPEC_CHKS 48) ]) (define_constants @@ -183,6 +186,10 @@ (define_attr "control_speculative" "no,yes" (const_string "no")) (define_attr "check_load" "no,yes" (const_string "no")) + +(define_attr "speculable1" "no,yes" (const_string "no")) + +(define_attr "speculable2" "no,yes" (const_string "no")) ;; DFA descriptions of ia64 processors used for insn scheduling and ;; bundling. @@ -232,7 +239,9 @@ ld1%O1 %0 = %1%P1 st1%Q0 %0 = %1%P0 mov %0 = %1" - [(set_attr "itanium_class" "icmp,icmp,unknown,unknown,tbit,ialu,ld,st,ialu")]) + [(set_attr "itanium_class" "icmp,icmp,unknown,unknown,tbit,ialu,ld,st,ialu") + (set_attr "speculable1" "yes") + (set_attr "speculable2" "no, no, no, no, no, no, yes,no,no")]) (define_split [(set (match_operand:BI 0 "register_operand" "") @@ -271,7 +280,7 @@ operands[1] = op1; }) -(define_insn "*movqi_internal" +(define_insn "movqi_internal" [(set (match_operand:QI 0 "destination_operand" "=r,r,r, m, r,*f,*f") (match_operand:QI 1 "move_operand" "rO,J,m,rO,*f,rO,*f"))] "ia64_move_ok (operands[0], operands[1])" @@ -283,7 +292,9 @@ getf.sig %0 = %1 setf.sig %0 = %r1 mov %0 = %1" - [(set_attr "itanium_class" "ialu,ialu,ld,st,frfr,tofr,fmisc")]) + [(set_attr "itanium_class" "ialu,ialu,ld,st,frfr,tofr,fmisc") + (set_attr "speculable1" "yes") + (set_attr "speculable2" "no, no, yes,no,no, no, no")]) (define_expand "movhi" [(set (match_operand:HI 0 "general_operand" "") @@ -296,7 +307,7 @@ operands[1] = op1; }) -(define_insn "*movhi_internal" +(define_insn "movhi_internal" [(set (match_operand:HI 0 "destination_operand" "=r,r,r, m, r,*f,*f") (match_operand:HI 1 "move_operand" "rO,J,m,rO,*f,rO,*f"))] "ia64_move_ok (operands[0], operands[1])" @@ -308,7 +319,9 @@ getf.sig %0 = %1 setf.sig %0 = %r1 mov %0 = %1" - [(set_attr "itanium_class" "ialu,ialu,ld,st,frfr,tofr,fmisc")]) + [(set_attr "itanium_class" "ialu,ialu,ld,st,frfr,tofr,fmisc") + (set_attr "speculable1" "yes") + (set_attr "speculable2" "no, no, yes,no,no, no, no")]) (define_expand "movsi" [(set (match_operand:SI 0 "general_operand" "") @@ -321,7 +334,7 @@ operands[1] = op1; }) -(define_insn "*movsi_internal" +(define_insn "movsi_internal" [(set (match_operand:SI 0 "destination_operand" "=r,r,r,r, m, r,*f,*f, r,*d") (match_operand:SI 1 "move_operand" "rO,J,i,m,rO,*f,rO,*f,*d,rK"))] "ia64_move_ok (operands[0], operands[1])" @@ -337,7 +350,9 @@ mov %0 = %1 mov %0 = %r1" ;; frar_m, toar_m ??? why not frar_i and toar_i - [(set_attr "itanium_class" "ialu,ialu,long_i,ld,st,frfr,tofr,fmisc,frar_m,toar_m")]) + [(set_attr "itanium_class" "ialu,ialu,long_i,ld,st,frfr,tofr,fmisc,frar_m,toar_m") + (set_attr "speculable1" "yes") + (set_attr "speculable2" "no, no, no, yes,no,no, no, no, no, no")]) (define_expand "movdi" [(set (match_operand:DI 0 "general_operand" "") @@ -350,7 +365,7 @@ operands[1] = op1; }) -(define_insn "*movdi_internal" +(define_insn "movdi_internal" [(set (match_operand:DI 0 "destination_operand" "=r,r,r,r, m, r,*f,*f,*f, Q, r,*b, r,*e, r,*d, r,*c") (match_operand:DI 1 "move_operand" @@ -383,7 +398,9 @@ return alt[which_alternative]; } - [(set_attr "itanium_class" "ialu,ialu,long_i,ld,st,frfr,tofr,fmisc,fld,stf,frbr,tobr,frar_i,toar_i,frar_m,toar_m,frpr,topr")]) + [(set_attr "itanium_class" "ialu,ialu,long_i,ld,st,frfr,tofr,fmisc,fld,stf,frbr,tobr,frar_i,toar_i,frar_m,toar_m,frpr,topr") + (set_attr "speculable1" "yes") + (set_attr "speculable2" "no, no, no, yes,no,no, no, no, yes,no, no, no, no, no, no, no, no, no")]) (define_mode_macro MODE [BI QI HI SI DI SF DF XF TI]) (define_mode_macro MODE_FOR_EXTEND [QI HI SI]) @@ -468,6 +485,26 @@ (XF "ldfe.c.clr %0 = %1%P1") (TI "ldfp8.c.clr %X0 = %1%P1")]) +(define_mode_attr output_c_nc [ + (BI "ld1.c.nc%O1 %0 = %1%P1") + (QI "ld1.c.nc%O1 %0 = %1%P1") + (HI "ld2.c.nc%O1 %0 = %1%P1") + (SI "ld4.c.nc%O1 %0 = %1%P1") + (DI + "@ + ld8.c.nc%O1 %0 = %1%P1 + ldf8.c.nc %0 = %1%P1") + (SF + "@ + ldfs.c.nc %0 = %1%P1 + ld4.c.nc%O1 %0 = %1%P1") + (DF + "@ + ldfd.c.nc %0 = %1%P1 + ld8.c.nc%O1 %0 = %1%P1") + (XF "ldfe.c.nc %0 = %1%P1") + (TI "ldfp8.c.nc %X0 = %1%P1")]) + (define_mode_attr ld_reg_constr [(BI "=*r") (QI "=r") (HI "=r") (SI "=r") (DI "=r,*f") (SF "=f,*r") (DF "=f,*r") (XF "=f") (TI "=*x")]) (define_mode_attr ldc_reg_constr [(BI "+*r") (QI "+r") (HI "+r") (SI "+r") (DI "+r,*f") (SF "+f,*r") (DF "+f,*r") (XF "+f") (TI "+*x")]) (define_mode_attr chk_reg_constr [(BI "*r") (QI "r") (HI "r") (SI "r") (DI "r,*f") (SF "f,*r") (DF "f,*r") (XF "f") (TI "*x")]) @@ -523,6 +560,15 @@ (set_attr "data_speculative" "<attr_yes>") (set_attr "control_speculative" "<attr_yes>")]) +(define_insn "mov<mode>_speculative_a" + [(set (match_operand:MODE 0 "<reg_pred_prefix>_register_operand" "<ld_reg_constr>") + (unspec:MODE [(match_operand:MODE 1 "memory_operand" "<mem_constr>")] UNSPEC_LDS_A))] + "ia64_move_ok (operands[0], operands[1])" + "<output_sa>" + [(set_attr "itanium_class" "<ld_class>") + (set_attr "data_speculative" "<attr_yes>") + (set_attr "control_speculative" "<attr_yes>")]) + (define_insn "zero_extend<mode>di2_speculative_advanced" [(set (match_operand:DI 0 "gr_register_operand" "=r") (zero_extend:DI (unspec:MODE_FOR_EXTEND [(match_operand:MODE_FOR_EXTEND 1 "memory_operand" "<mem_constr>")] UNSPEC_LDSA)))] @@ -532,6 +578,15 @@ (set_attr "data_speculative" "<attr_yes>") (set_attr "control_speculative" "<attr_yes>")]) +(define_insn "zero_extend<mode>di2_speculative_a" + [(set (match_operand:DI 0 "gr_register_operand" "=r") + (zero_extend:DI (unspec:MODE_FOR_EXTEND [(match_operand:MODE_FOR_EXTEND 1 "memory_operand" "<mem_constr>")] UNSPEC_LDS_A)))] + "" + "<output_sa>" + [(set_attr "itanium_class" "<ld_class>") + (set_attr "data_speculative" "<attr_yes>") + (set_attr "control_speculative" "<attr_yes>")]) + (define_insn "mov<mode>_clr" [(set (match_operand:MODE 0 "<reg_pred_prefix>_register_operand" "<ldc_reg_constr>") (if_then_else:MODE (ne (unspec [(match_dup 0)] UNSPEC_LDCCLR) (const_int 0)) @@ -542,6 +597,16 @@ [(set_attr "itanium_class" "<ld_class>") (set_attr "check_load" "<attr_yes>")]) +(define_insn "mov<mode>_nc" + [(set (match_operand:MODE 0 "<reg_pred_prefix>_register_operand" "<ldc_reg_constr>") + (if_then_else:MODE (ne (unspec [(match_dup 0)] UNSPEC_LDCNC) (const_int 0)) + (match_operand:MODE 1 "memory_operand" "<mem_constr>") + (match_dup 0)))] + "ia64_move_ok (operands[0], operands[1])" + "<output_c_nc>" + [(set_attr "itanium_class" "<ld_class>") + (set_attr "check_load" "<attr_yes>")]) + (define_insn "zero_extend<mode>di2_clr" [(set (match_operand:DI 0 "gr_register_operand" "+r") (if_then_else:DI (ne (unspec [(match_dup 0)] UNSPEC_LDCCLR) (const_int 0)) @@ -552,6 +617,16 @@ [(set_attr "itanium_class" "<ld_class>") (set_attr "check_load" "<attr_yes>")]) +(define_insn "zero_extend<mode>di2_nc" + [(set (match_operand:DI 0 "gr_register_operand" "+r") + (if_then_else:DI (ne (unspec [(match_dup 0)] UNSPEC_LDCNC) (const_int 0)) + (zero_extend:DI (match_operand:MODE_FOR_EXTEND 1 "memory_operand" "<mem_constr>")) + (match_dup 0)))] + "" + "<output_c_nc>" + [(set_attr "itanium_class" "<ld_class>") + (set_attr "check_load" "<attr_yes>")]) + (define_insn "advanced_load_check_clr_<mode>" [(set (pc) (if_then_else (ne (unspec [(match_operand:MODE 0 "<reg_pred_prefix>_register_operand" "<chk_reg_constr>")] UNSPEC_CHKACLR) (const_int 0)) @@ -561,6 +636,15 @@ "chk.a.clr %0, %l1" [(set_attr "itanium_class" "<chka_class>")]) +(define_insn "advanced_load_check_nc_<mode>" + [(set (pc) + (if_then_else (ne (unspec [(match_operand:MODE 0 "<reg_pred_prefix>_register_operand" "<chk_reg_constr>")] UNSPEC_CHKANC) (const_int 0)) + (pc) + (label_ref (match_operand 1 "" ""))))] + "" + "chk.a.clr %0, %l1" + [(set_attr "itanium_class" "<chka_class>")]) + (define_insn "speculation_check_<mode>" [(set (pc) (if_then_else (ne (unspec [(match_operand:MODE 0 "<reg_pred_prefix>_register_operand" "<chk_reg_constr>")] UNSPEC_CHKS) (const_int 0)) @@ -856,7 +940,7 @@ operands[1] = op1; }) -(define_insn_and_split "*movti_internal" +(define_insn_and_split "movti_internal" [(set (match_operand:TI 0 "destination_operand" "=r, *fm,*x,*f, Q") (match_operand:TI 1 "general_operand" "r*fim,r, Q, *fOQ,*f"))] "ia64_move_ok (operands[0], operands[1])" @@ -872,7 +956,9 @@ ia64_split_tmode_move (operands); DONE; } - [(set_attr "itanium_class" "unknown,unknown,fldp,unknown,unknown")]) + [(set_attr "itanium_class" "unknown,unknown,fldp,unknown,unknown") + (set_attr "speculable1" "yes") + (set_attr "speculable2" "no, no, yes, no, no")]) ;; Floating Point Moves ;; @@ -890,7 +976,7 @@ operands[1] = op1; }) -(define_insn "*movsf_internal" +(define_insn "movsf_internal" [(set (match_operand:SF 0 "destination_operand" "=f,f, Q,*r, f,*r,*r, m") (match_operand:SF 1 "general_operand" "fG,Q,fG,fG,*r,*r, m,*r"))] "ia64_move_ok (operands[0], operands[1])" @@ -903,7 +989,9 @@ mov %0 = %1 ld4%O1 %0 = %1%P1 st4%Q0 %0 = %1%P0" - [(set_attr "itanium_class" "fmisc,fld,stf,frfr,tofr,ialu,ld,st")]) + [(set_attr "itanium_class" "fmisc,fld,stf,frfr,tofr,ialu,ld,st") + (set_attr "speculable1" "yes") + (set_attr "speculable2" "no, yes,no, no, no, no, yes,no")]) (define_expand "movdf" [(set (match_operand:DF 0 "general_operand" "") @@ -916,7 +1004,7 @@ operands[1] = op1; }) -(define_insn "*movdf_internal" +(define_insn "movdf_internal" [(set (match_operand:DF 0 "destination_operand" "=f,f, Q,*r, f,*r,*r, m") (match_operand:DF 1 "general_operand" "fG,Q,fG,fG,*r,*r, m,*r"))] "ia64_move_ok (operands[0], operands[1])" @@ -929,7 +1017,9 @@ mov %0 = %1 ld8%O1 %0 = %1%P1 st8%Q0 %0 = %1%P0" - [(set_attr "itanium_class" "fmisc,fld,stf,frfr,tofr,ialu,ld,st")]) + [(set_attr "itanium_class" "fmisc,fld,stf,frfr,tofr,ialu,ld,st") + (set_attr "speculable1" "yes") + (set_attr "speculable2" "no, yes,no, no, no, no, yes,no")]) ;; With no offsettable memory references, we've got to have a scratch ;; around to play with the second word if the variable winds up in GRs. @@ -944,7 +1034,7 @@ ;; ??? There's no easy way to mind volatile acquire/release semantics. -(define_insn "*movxf_internal" +(define_insn "movxf_internal" [(set (match_operand:XF 0 "destination_operand" "=f,f, m") (match_operand:XF 1 "general_operand" "fG,m,fG"))] "ia64_move_ok (operands[0], operands[1])" @@ -952,7 +1042,9 @@ mov %0 = %F1 ldfe %0 = %1%P1 stfe %0 = %F1%P0" - [(set_attr "itanium_class" "fmisc,fld,stf")]) + [(set_attr "itanium_class" "fmisc,fld,stf") + (set_attr "speculable1" "yes") + (set_attr "speculable2" "no, yes,no")]) ;; Same as for movxf, but for RFmode. (define_expand "movrf" @@ -1042,7 +1134,9 @@ "@ zxt1 %0 = %1 ld1%O1 %0 = %1%P1" - [(set_attr "itanium_class" "xtd,ld")]) + [(set_attr "itanium_class" "xtd,ld") + (set_attr "speculable1" "yes") + (set_attr "speculable2" "no, yes")]) (define_insn "zero_extendhidi2" [(set (match_operand:DI 0 "gr_register_operand" "=r,r") @@ -1051,7 +1145,9 @@ "@ zxt2 %0 = %1 ld2%O1 %0 = %1%P1" - [(set_attr "itanium_class" "xtd,ld")]) + [(set_attr "itanium_class" "xtd,ld") + (set_attr "speculable1" "yes") + (set_attr "speculable2" "no, yes")]) (define_insn "zero_extendsidi2" [(set (match_operand:DI 0 "grfr_register_operand" "=r,r,?f") @@ -1062,7 +1158,9 @@ addp4 %0 = %1, r0 ld4%O1 %0 = %1%P1 fmix.r %0 = f0, %1" - [(set_attr "itanium_class" "ialu,ld,fmisc")]) + [(set_attr "itanium_class" "ialu,ld,fmisc") + (set_attr "speculable1" "yes") + (set_attr "speculable2" "no, yes,no")]) ;; Convert between floating point types of different sizes. diff --git a/gcc/config/ia64/ia64.opt b/gcc/config/ia64/ia64.opt index 08097e754ac..91c88da5849 100644 --- a/gcc/config/ia64/ia64.opt +++ b/gcc/config/ia64/ia64.opt @@ -120,12 +120,12 @@ msched-in-control-spec Target Report Var(mflag_sched_in_control_spec) Init(1) Use in block control speculation -msched-ldc -Target Report Var(mflag_sched_ldc) Init(1) +msched-spec-ldc +Target Report Var(mflag_sched_spec_ldc) Init(1) Use simple data speculation check -msched-control-ldc -Target Report Var(mflag_control_ldc) Init(0) +msched-spec-control-ldc +Target Report Var(mflag_sched_spec_control_ldc) Init(0) Use simple data speculation check for control speculation msched-spec-verbose @@ -160,4 +160,16 @@ msel-sched-substitution Common Report Var(mflag_sel_sched_substitution) Init(1) Perform substitution in selective scheduling +msel-sched-data-spec +Common Report Var(mflag_sel_sched_data_spec) Init(0) +Perform data speculation in selective scheduling + +msel-sched-control-spec +Common Report Var(mflag_sel_sched_control_spec) Init(1) +Perform control speculation in selective scheduling + +msel-sched-dont-check-control-spec +Common Report Var(mflag_sel_sched_dont_check_control_spec) Init(0) +Don't generate checks for control speculation in selective scheduling + ; This comment is to ensure we retain the blank line above. diff --git a/gcc/emit-rtl.c b/gcc/emit-rtl.c index becde4f9e4f..1ad5c2bb87a 100644 --- a/gcc/emit-rtl.c +++ b/gcc/emit-rtl.c @@ -3928,6 +3928,10 @@ emit_insn_after_1 (rtx first, rtx after) if (after == last_insn) last_insn = last; + + if (insn_added) + insn_added (last); + return last; } diff --git a/gcc/haifa-sched.c b/gcc/haifa-sched.c index ce1a0d93091..f66c42e1a06 100644 --- a/gcc/haifa-sched.c +++ b/gcc/haifa-sched.c @@ -145,6 +145,7 @@ Software Foundation, 51 Franklin Street, Fifth Floor, Boston, MA #include "target.h" #include "output.h" #include "params.h" +#include "vecprim.h" #ifdef INSN_SCHEDULING @@ -170,9 +171,6 @@ int sched_verbose = 0; either to stderr, or to the dump listing file (-dRS). */ FILE *sched_dump = 0; -/* Current highest uid. */ -int sched_max_uid = 0; - /* fix_sched_param() is called from toplev.c upon detection of the -fsched-verbose=N option. */ @@ -185,21 +183,12 @@ fix_sched_param (const char *param, const char *val) warning (0, "fix_sched_param: unknown param: %s", param); } -struct haifa_insn_data *h_i_d; - /* This is a placeholder for the scheduler parameters common to all schedulers. */ struct common_sched_info_def *common_sched_info; -/* Mapping from instruction UID to its Logical UID. */ -int *uid_to_luid = NULL; - -/* This weight is an estimation of the insn's contribution to - register pressure. */ -short *reg_weights = NULL; - -#define INSN_TICK(INSN) (h_i_d[INSN_UID (INSN)].tick) -#define INTER_TICK(INSN) (h_i_d[INSN_UID (INSN)].inter_tick) +#define INSN_TICK(INSN) (HID (INSN)->tick) +#define INTER_TICK(INSN) (HID (INSN)->inter_tick) /* If INSN_TICK of an instruction is equal to INVALID_TICK, then it should be recalculated from scratch. */ @@ -215,16 +204,10 @@ short *reg_weights = NULL; last element in the list. */ rtx note_list; -/* Blocks in the scheduling region. */ -sbitmap sched_bbs = NULL; - -/* Insns in the scheduling region. */ -sbitmap sched_insns = NULL; - static struct spec_info_def spec_info_var; /* Description of the speculative part of the scheduling. If NULL - no speculation. */ -static spec_info_t spec_info; +spec_info_t spec_info; /* True, if recovery block was added during scheduling of current block. Used to determine, if we need to fix INSN_TICKs. */ @@ -233,15 +216,9 @@ static bool added_recovery_block_p; /* Counters of different types of speculative instructions. */ static int nr_begin_data, nr_be_in_data, nr_begin_control, nr_be_in_control; -/* Pointers to GLAT data. See glat_init for more information. */ -regset *glat_start = NULL, *glat_end = NULL; - /* Array used in {unlink, restore}_bb_notes. */ static rtx *bb_header = 0; -/* Index of the current last_basic_block. */ -int sched_last_basic_block = 0; - /* Basic block after which recovery blocks will be created. */ static basic_block before_recovery; @@ -305,7 +282,7 @@ static int q_size = 0; QUEUE_READY - INSN is in ready list. N >= 0 - INSN queued for X [where NEXT_Q_AFTER (q_ptr, X) == N] cycles. */ -#define QUEUE_INDEX(INSN) (h_i_d[INSN_UID (INSN)].queue_index) +#define QUEUE_INDEX(INSN) (HID (INSN)->queue_index) /* The following variable value refers for all current and future reservations of the processor units. */ @@ -328,9 +305,6 @@ static struct ready_list *readyp = &ready; /* Scheduling clock. */ static int clock_var; -/* Number of instructions in current scheduling region. */ -static int rgn_n_insns = -1; - static int may_trap_exp (rtx, int); /* Nonzero iff the address is comprised from at most 1 register. */ @@ -543,26 +517,20 @@ static void resolve_dep (rtx, rtx); /* The following functions are used to implement scheduling of data/control speculative instructions. */ -static void extend_ready (int); -static void haifa_insns_init (rtx); -static void extend_all (rtx); static void generate_recovery_code (rtx); static void process_insn_depend_be_in_spec (rtx, rtx, ds_t); static void begin_speculative_block (rtx); static void add_to_speculative_block (rtx); -static dw_t dep_weak (ds_t); static void init_before_recovery (void); static basic_block create_recovery_block (void); static void create_check_block_twin (rtx, bool); static void fix_recovery_deps (basic_block); -static void change_pattern (rtx, rtx); -static int speculate_insn (rtx, ds_t, rtx *); +static void haifa_change_pattern (rtx, rtx); static void dump_new_block_header (int, basic_block, rtx, rtx); static void restore_bb_notes (basic_block); static void fix_jump_move (rtx); static void move_block_after_check (rtx); static void move_succs (VEC(edge,gc) **, basic_block); -static void glat_init (basic_block); static void attach_life_info1 (basic_block); static void sched_remove_insn (rtx); static void clear_priorities (rtx); @@ -586,9 +554,6 @@ schedule_insns (void) } #else -/* Working copy of frontend's sched_info variable. */ -static struct haifa_sched_info current_sched_info_var; - /* Pointer to the last instruction scheduled. Used by rank_for_schedule, so that insns independent of the last scheduled insn will be preferred over dependent instructions. */ @@ -851,13 +816,13 @@ rank_for_schedule (const void *x, const void *y) ds1 = TODO_SPEC (tmp) & SPECULATIVE; if (ds1) - dw1 = dep_weak (ds1); + dw1 = ds_weak (ds1); else dw1 = NO_DEP_WEAK; ds2 = TODO_SPEC (tmp2) & SPECULATIVE; if (ds2) - dw2 = dep_weak (ds2); + dw2 = ds_weak (ds2); else dw2 = NO_DEP_WEAK; @@ -962,7 +927,7 @@ queue_insn (rtx insn, int n_cycles) fprintf (sched_dump, "queued for %d cycles.\n", n_cycles); } - + QUEUE_INDEX (insn) = next_q; } @@ -2687,9 +2652,6 @@ set_priorities (rtx head, rtx tail) return n_insn; } -/* Next LUID to assign to an instruction. */ -int max_luid = 1; - /* Initialize some global state for the scheduler. This function works with the common data shared between all the schedulers. It is called from the scheduler specific initialization routine. */ @@ -2716,12 +2678,14 @@ sched_init (void) { spec_info = &spec_info_var; targetm.sched.set_sched_flags (spec_info); - if (sched_deps_info->generate_spec_deps) + + if (spec_info->mask != 0) spec_info->weakness_cutoff = (PARAM_VALUE (PARAM_SCHED_SPEC_PROB_CUTOFF) * MAX_DEP_WEAK) / 100; else /* So we won't read anything accidentally. */ - spec_info = 0; + spec_info = NULL; + #ifdef ENABLE_CHECKING check_sched_flags (); #endif @@ -2760,13 +2724,8 @@ sched_init (void) init_alias_analysis (); if (targetm.sched.md_init_global) - targetm.sched.md_init_global (sched_dump, sched_verbose, sched_max_uid); - - /* GLAT_{START, END} [Global Live AT {START, END}] should be a region-scope - data structure, but we would like to use it for global register check - in sched-rgn.c: sched_rgn_global_finish (). For this cause it is - initialized and freed on the most outer level. */ - glat_init (NULL); + targetm.sched.md_init_global (sched_dump, sched_verbose, + get_max_uid () + 1); curr_state = xmalloc (dfa_state_size); } @@ -2777,12 +2736,36 @@ haifa_sched_init (void) { sched_init (); - /* Switch to working copy of sched_info. */ - memcpy (¤t_sched_info_var, current_sched_info, - sizeof (current_sched_info_var)); - current_sched_info = ¤t_sched_info_var; - - sched_bbs_init (NULL); + if (spec_info != NULL) + { + gcc_assert (common_sched_info->use_glat); + + common_sched_info->detach_life_info = 1; + sched_deps_info->use_deps_list = 1; + sched_deps_info->generate_spec_deps = 1; + } + + /* Initialize glat, luids, dependency caches, target and h_i_d for the + whole function. */ + { + bb_vec_t bbs = VEC_alloc (basic_block, heap, n_basic_blocks); + basic_block bb; + + FOR_ALL_BB (bb) + VEC_quick_push (basic_block, bbs, bb); + + sched_init_bbs (bbs, NULL); + + VEC_pop (basic_block, bbs); + VEC_ordered_remove (basic_block, bbs, 0); + + sched_init_luids (bbs, NULL, NULL, NULL); + sched_deps_local_init (true); + sched_extend_target (); + haifa_init_h_i_d (bbs, NULL, NULL, NULL); + + VEC_free (basic_block, heap, bbs); + } #ifdef ENABLE_CHECKING /* This is used preferably for finding bugs in check_cfg () itself. @@ -2791,8 +2774,6 @@ haifa_sched_init (void) check_cfg (0, 0); #endif - haifa_insns_init (NULL); - nr_begin_data = nr_begin_control = nr_be_in_data = nr_be_in_control = 0; before_recovery = 0; } @@ -2822,11 +2803,14 @@ haifa_sched_finish (void) c, nr_be_in_control); } - /* haifa_insns_finish. */ - free (h_i_d); - h_i_d = NULL; - sched_insns_finish (); - sched_bbs_finish (); + /* Finalize h_i_d, dependency caches, luids and glat for the whole + function. Target will be finalized in md_global_finish (). */ + { + haifa_finish_h_i_d (); + sched_deps_local_finish (); + sched_finish_luids (); + sched_finish_bbs (); + } current_sched_info = NULL; @@ -2839,42 +2823,15 @@ haifa_sched_finish (void) void sched_finish (void) { - if (common_sched_info->use_glat) - { - /* Free GLAT information that was created by attach_life_info () - and used by sched-rgn.c: sched_rgn_global_finish (). */ -#ifdef ENABLE_CHECKING - if (common_sched_info->detach_life_info) - { - basic_block bb; - - FOR_ALL_BB (bb) - { - int i = bb->index; - - if (glat_start[i]) - FREE_REG_SET (glat_start[i]); - if (glat_end[i]) - FREE_REG_SET (glat_end[i]); - } - } -#endif - - free (glat_start); - glat_start = NULL; - - free (glat_end); - glat_end = NULL; - } - free (curr_state); - dfa_finish (); - end_alias_analysis (); - if (targetm.sched.md_finish_global) targetm.sched.md_finish_global (sched_dump, sched_verbose); + end_alias_analysis (); + + dfa_finish (); + #ifdef ENABLE_CHECKING /* After reload ia64 backend clobbers CFG, so can't check anything. */ if (!reload_completed) @@ -2950,6 +2907,8 @@ fix_inter_tick (rtx head, rtx tail) } bitmap_clear (&processed); } + +static int haifa_speculate_insn (rtx, ds_t, rtx *); /* Check if NEXT is ready to be added to the ready or queue list. If "yes", add it to the proper list. @@ -2994,7 +2953,7 @@ try_ready (rtx next) while ((link = XEXP (link, 1))) *ts = ds_merge (*ts, DEP_STATUS (link) & SPECULATIVE); - if (dep_weak (*ts) < spec_info->weakness_cutoff) + if (ds_weak (*ts) < spec_info->weakness_cutoff) /* Too few points. */ *ts = (*ts & ~SPECULATIVE) | HARD_DEP; } @@ -3029,7 +2988,7 @@ try_ready (rtx next) gcc_assert ((*ts & SPECULATIVE) && !(*ts & ~SPECULATIVE)); - res = speculate_insn (next, *ts, &new_pat); + res = haifa_speculate_insn (next, *ts, &new_pat); switch (res) { @@ -3053,7 +3012,7 @@ try_ready (rtx next) save it. */ ORIG_PAT (next) = PATTERN (next); - change_pattern (next, new_pat); + haifa_change_pattern (next, new_pat); break; default: @@ -3084,7 +3043,7 @@ try_ready (rtx next) ORIG_PAT field. Except one case - speculation checks have ORIG_PAT pat too, so skip them. */ { - change_pattern (next, ORIG_PAT (next)); + haifa_change_pattern (next, ORIG_PAT (next)); ORIG_PAT (next) = 0; } @@ -3218,55 +3177,46 @@ resolve_dep (rtx next, rtx insn) && (LOG_LINKS (next) || INSN_DEP_COUNT (next) == 0)); } -/* Extend READY, READY_TRY and CHOICE_STACK arrays. - N_NEW_INSNS is the number of additional elements to allocate. */ -static void -extend_ready (int n_new_insns) -{ - if (rgn_n_insns == -1) - /* Compensate fake insn. */ - n_new_insns++; - - ready.veclen = rgn_n_insns + n_new_insns + issue_rate; - ready.vec = XRESIZEVEC (rtx, ready.vec, ready.veclen); - - sched_local_init (rgn_n_insns + n_new_insns); -} - -/* Initialize per region data structures for the haifa scheduler. */ -void -haifa_local_init (int n_insns) -{ - extend_ready (n_insns); -} +static int sched_ready_n_insns = -1; /* Initialize per region data structures. */ void -sched_local_init (int n_insns) +sched_extend_ready_list (int new_sched_ready_n_insns) { - int i = rgn_n_insns + 1; + int i; + + if (sched_ready_n_insns == -1) + /* At the first call we need to initialize one more choice_stack + entry. */ + { + i = 0; + sched_ready_n_insns = 0; + } + else + i = sched_ready_n_insns + 1; - /* We initially set RGN_N_INSNS to -1 to avoid memory leak in allocating - +1 element for choice_stack[i].state . */ - rgn_n_insns = 0; + ready.veclen = new_sched_ready_n_insns + issue_rate; + ready.vec = XRESIZEVEC (rtx, ready.vec, ready.veclen); - gcc_assert (n_insns >= rgn_n_insns); + gcc_assert (new_sched_ready_n_insns >= sched_ready_n_insns); - ready_try = xrecalloc (ready_try, n_insns, rgn_n_insns, sizeof (*ready_try)); + ready_try = xrecalloc (ready_try, new_sched_ready_n_insns, + sched_ready_n_insns, sizeof (*ready_try)); /* We allocate +1 element to save initial state in the choice_stack[0] entry. */ - choice_stack = XRESIZEVEC (struct choice_entry, choice_stack, n_insns + 1); + choice_stack = XRESIZEVEC (struct choice_entry, choice_stack, + new_sched_ready_n_insns + 1); - for (; i <= n_insns; i++) + for (; i <= new_sched_ready_n_insns; i++) choice_stack[i].state = xmalloc (dfa_state_size); - rgn_n_insns = n_insns; + sched_ready_n_insns = new_sched_ready_n_insns; } /* Free per region data structures. */ void -sched_local_finish (void) +sched_finish_ready_list (void) { int i; @@ -3277,156 +3227,35 @@ sched_local_finish (void) free (ready_try); ready_try = NULL; - for (i = 0; i <= rgn_n_insns; i++) + for (i = 0; i <= sched_ready_n_insns; i++) free (choice_stack [i].state); free (choice_stack); choice_stack = NULL; - rgn_n_insns = -1; + sched_ready_n_insns = -1; } -/* Init per-instruction data for INSN. If INSN is NULL, perform - initialization for all insns of the blocks in the sched_bbs bitmap. */ -void -sched_insns_init (rtx insn) -{ - int sched_max_uid1 = get_max_uid () + 1; - - uid_to_luid = xrealloc (uid_to_luid, sched_max_uid1 * sizeof (*uid_to_luid)); - - if (insn) - INSN_LUID (insn) = max_luid++; - else - { - unsigned int i; - sbitmap_iterator sbi; - - gcc_assert (max_luid == 1); - - uid_to_luid[0] = 0; - - EXECUTE_IF_SET_IN_SBITMAP (sched_bbs, 0, i, sbi) - { - basic_block bb = BASIC_BLOCK (i); - rtx x; - - for (x = BB_HEAD (bb); ; x = NEXT_INSN (x)) - { - INSN_LUID (x) = max_luid; - - /* Increment the next luid, unless this is a note. We don't - really need separate IDs for notes and we don't want to - schedule differently depending on whether or not there are - line-number notes, i.e., depending on whether or not we're - generating debugging information. */ - if (!NOTE_P (x)) - ++max_luid; - - if (x == BB_END (bb)) - break; - } - } - } - - sched_deps_local_init (insn == NULL); - - reg_weights = xrealloc (reg_weights, max_luid * sizeof (*reg_weights)); - if (insn) - find_insn_reg_weight (insn); - else - { - unsigned int i; - sbitmap_iterator sbi; - - EXECUTE_IF_SET_IN_SBITMAP (sched_bbs, 0, i, sbi) - { - basic_block bb = BASIC_BLOCK (i); - rtx x = BB_HEAD (bb); - - do - { - find_insn_reg_weight (x); - - if (x == BB_END (bb)) - break; - - x = NEXT_INSN (x); - } - while (1); - } - } - - sched_max_uid = sched_max_uid1; -} - -/* Free all per-instruction data. */ -void -sched_insns_finish (void) -{ - free (reg_weights); - reg_weights = NULL; - - sched_deps_local_finish (); - - free (uid_to_luid); - uid_to_luid = NULL; - - max_luid = 1; - - sched_max_uid = 0; -} - -/* Extend global scheduler structures (those, that live across calls to - schedule_block) to include information about just emitted INSN. */ -static void -haifa_insns_init (rtx insn) +static int +haifa_luid_for_non_insn (rtx x) { - int old_sched_max_uid = sched_max_uid; - - sched_insns_init (insn); - - h_i_d = xrecalloc (h_i_d, sched_max_uid, old_sched_max_uid, sizeof (*h_i_d)); + gcc_assert (NOTE_P (x) || LABEL_P (x)); - /* Initialize h_i_d entry of the new INSN with default values. - Values, that are not explicitly initialized here, hold zero. */ - if (insn) - { - INSN_COST (insn) = -1; - TODO_SPEC (insn) = HARD_DEP; - QUEUE_INDEX (insn) = QUEUE_NOWHERE; - INSN_TICK (insn) = INVALID_TICK; - INTER_TICK (insn) = INVALID_TICK; - } - else - { - int i; - - for (i = 0; i < sched_max_uid; i++) - { - h_i_d[i].cost = -1; - h_i_d[i].todo_spec = HARD_DEP; - h_i_d[i].queue_index = QUEUE_NOWHERE; - h_i_d[i].tick = INVALID_TICK; - h_i_d[i].inter_tick = INVALID_TICK; - } - } - - if (targetm.sched.h_i_d_extended) - targetm.sched.h_i_d_extended (); + return 0; } -/* Extends global and local scheduler structures to include information - about just emitted INSN. */ -static void -extend_all (rtx insn) -{ - haifa_insns_init (insn); - - /* These structures have block scope. */ - extend_ready (1); - - (*current_sched_info->add_remove_insn) (insn, 0); -} +const struct common_sched_info_def haifa_common_sched_info = + { + NULL, /* fix_recovery_cfg */ + NULL, /* add_block */ + NULL, /* estimate_number_of_insns */ + haifa_luid_for_non_insn, /* luid_for_non_insn */ +#ifdef ENABLE_CHECKING + NULL, /* region_head_or_leaf_p */ +#endif + 0, /* use_glat */ + 0, /* detach_life_info */ + SCHED_PASS_UNKNOWN /* sched_pass_id */ + }; /* Generates recovery code for INSN. */ static void @@ -3473,7 +3302,7 @@ process_insn_depend_be_in_spec (rtx link, rtx twin, ds_t fs) it can be removed from the ready (or queue) list only due to backend decision. Hence we can't let the probability of the speculative dep to decrease. */ - dep_weak (ds) <= dep_weak (fs)) + ds_weak (ds) <= ds_weak (fs)) /* Transform it to be in speculative. */ ds = (ds & ~BEGIN_SPEC) | fs; } @@ -3500,6 +3329,8 @@ begin_speculative_block (rtx insn) TODO_SPEC (insn) &= ~BEGIN_SPEC; } +static void haifa_init_insn (rtx); + /* Generates recovery code for BE_IN speculative INSN. */ static void add_to_speculative_block (rtx insn) @@ -3556,7 +3387,7 @@ add_to_speculative_block (rtx insn) rec = BLOCK_FOR_INSN (check); twin = emit_insn_before (copy_rtx (PATTERN (insn)), BB_END (rec)); - haifa_insns_init (twin); + haifa_init_insn (twin); RESOLVED_DEPS (twin) = copy_DEPS_LIST_list (RESOLVED_DEPS (insn)); @@ -3633,41 +3464,6 @@ xrecalloc (void *p, size_t new_nmemb, size_t old_nmemb, size_t size) return p; } -/* Return the probability of speculation success for the speculation - status DS. */ -static dw_t -dep_weak (ds_t ds) -{ - ds_t res = 1, dt; - int n = 0; - - dt = FIRST_SPEC_TYPE; - do - { - if (ds & dt) - { - res *= (ds_t) get_dep_weak (ds, dt); - n++; - } - - if (dt == LAST_SPEC_TYPE) - break; - dt <<= SPEC_TYPE_SHIFT; - } - while (1); - - gcc_assert (n); - while (--n) - res /= MAX_DEP_WEAK; - - if (res < MIN_DEP_WEAK) - res = MIN_DEP_WEAK; - - gcc_assert (res <= MAX_DEP_WEAK); - - return (dw_t) res; -} - /* Helper function. Find fallthru edge from PRED. */ edge @@ -3742,12 +3538,12 @@ init_before_recovery (void) x = emit_jump_insn_after (gen_jump (label), BB_END (single)); JUMP_LABEL (x) = label; LABEL_NUSES (label)++; - haifa_insns_init (x); + haifa_init_insn (x); emit_barrier_after (x); - add_block (empty, 0); - add_block (single, 0); + haifa_init_only_bb (empty, NULL); + haifa_init_only_bb (single, NULL); before_recovery = single; @@ -3803,14 +3599,24 @@ create_check_block_twin (rtx insn, bool mutate_p) basic_block rec; rtx label, check, twin, link; ds_t fs; + ds_t todo_spec; + + gcc_assert (ORIG_PAT (insn) != NULL_RTX); + + if (!mutate_p) + todo_spec= TODO_SPEC (insn); + else + { + gcc_assert (IS_SPECULATION_SIMPLE_CHECK_P (insn) + && (TODO_SPEC (insn) & SPECULATIVE) == 0); - gcc_assert (ORIG_PAT (insn) - && (!mutate_p - || (IS_SPECULATION_SIMPLE_CHECK_P (insn) - && !(TODO_SPEC (insn) & SPECULATIVE)))); + todo_spec = CHECK_SPEC (insn); + } + + todo_spec &= SPECULATIVE; /* Create recovery block. */ - if (mutate_p || targetm.sched.needs_block_p (insn)) + if (mutate_p || targetm.sched.needs_block_p (todo_spec)) { rec = create_recovery_block (); label = BB_HEAD (rec); @@ -3818,11 +3624,11 @@ create_check_block_twin (rtx insn, bool mutate_p) else { rec = EXIT_BLOCK_PTR; - label = 0; + label = NULL_RTX; } /* Emit CHECK. */ - check = targetm.sched.gen_check (insn, label, mutate_p); + check = targetm.sched.gen_spec_check (insn, label, todo_spec); if (rec != EXIT_BLOCK_PTR) { @@ -3838,7 +3644,15 @@ create_check_block_twin (rtx insn, bool mutate_p) check = emit_insn_before (check, insn); /* Extend data structures. */ - extend_all (check); + haifa_init_insn (check); + + /* CHECK is being added to current region. Extend ready list. */ + gcc_assert (sched_ready_n_insns != -1); + sched_extend_ready_list (sched_ready_n_insns + 1); + + if (current_sched_info->add_remove_insn) + current_sched_info->add_remove_insn (insn, 0); + RECOVERY_BLOCK (check) = rec; if (sched_verbose && spec_info->dump) @@ -3862,7 +3676,7 @@ create_check_block_twin (rtx insn, bool mutate_p) } twin = emit_insn_after (ORIG_PAT (insn), BB_END (rec)); - haifa_insns_init (twin); + haifa_init_insn (twin); if (sched_verbose && spec_info->dump) /* INSN_BB (insn) isn't determined for twin insns yet. @@ -3906,14 +3720,14 @@ create_check_block_twin (rtx insn, bool mutate_p) e = make_edge (first_bb, rec, edge_flags); - add_block (second_bb, first_bb); + haifa_init_only_bb (second_bb, first_bb); gcc_assert (NOTE_INSN_BASIC_BLOCK_P (BB_HEAD (second_bb))); label = block_label (second_bb); jump = emit_jump_insn_after (gen_jump (label), BB_END (rec)); JUMP_LABEL (jump) = label; LABEL_NUSES (label)++; - haifa_insns_init (jump); + haifa_init_insn (jump); if (BB_PARTITION (second_bb) != BB_PARTITION (rec)) /* Partition type is the same, if it is "unpartitioned". */ @@ -3937,7 +3751,7 @@ create_check_block_twin (rtx insn, bool mutate_p) make_single_succ_edge (rec, second_bb, edge_flags); - add_block (rec, EXIT_BLOCK_PTR); + haifa_init_only_bb (rec, EXIT_BLOCK_PTR); } /* Move backward dependences from INSN to CHECK and @@ -4123,19 +3937,28 @@ fix_recovery_deps (basic_block rec) add_jump_dependencies (insn, jump); } -/* Changes pattern of the INSN to NEW_PAT. */ -static void -change_pattern (rtx insn, rtx new_pat) +/* Change pattern of INSN to NEW_PAT. */ +void +sched_change_pattern (rtx insn, rtx new_pat) { int t; t = validate_change (insn, &PATTERN (insn), new_pat, 0); gcc_assert (t); + dfa_clear_single_insn_cache (insn); +} + +/* Change pattern of INSN to NEW_PAT. Invalidate cached haifa + instruction data. */ +static void +haifa_change_pattern (rtx insn, rtx new_pat) +{ + sched_change_pattern (insn, new_pat); + /* Invalidate INSN_COST, so it'll be recalculated. */ INSN_COST (insn) = -1; /* Invalidate INSN_TICK, so it'll be recalculated. */ INSN_TICK (insn) = INVALID_TICK; - dfa_clear_single_insn_cache (insn); } @@ -4143,21 +3966,21 @@ change_pattern (rtx insn, rtx new_pat) 0 - for speculation with REQUEST mode it is OK to use current instruction pattern, 1 - need to change pattern for *NEW_PAT to be speculative. */ -static int -speculate_insn (rtx insn, ds_t request, rtx *new_pat) +int +sched_speculate_insn (rtx insn, ds_t request, rtx *new_pat) { - gcc_assert (sched_deps_info->generate_spec_deps - && (request & SPECULATIVE)); + request = ds_get_speculation_types (request); if (!NONJUMP_INSN_P (insn) - || HAS_INTERNAL_DEP (insn) - || SCHED_GROUP_P (insn) - || side_effects_p (PATTERN (insn)) - || (request & spec_info->mask) != request) + || (request & spec_info->mask) != request) return -1; - - gcc_assert (!IS_SPECULATION_CHECK_P (insn)); + if (request != 0 + && !(request == BEGIN_CONTROL + && (spec_info->flags & SEL_SCHED_SPEC_DONT_CHECK_CONTROL)) + && side_effects_p (PATTERN (insn))) + return -1; + if (request & BE_IN_SPEC) { if (may_trap_p (PATTERN (insn))) @@ -4167,7 +3990,22 @@ speculate_insn (rtx insn, ds_t request, rtx *new_pat) return 0; } - return targetm.sched.speculate_insn (insn, request & BEGIN_SPEC, new_pat); + request &= BEGIN_SPEC; + + return targetm.sched.speculate_insn (insn, request, new_pat); +} + +static int +haifa_speculate_insn (rtx insn, ds_t request, rtx *new_pat) +{ + gcc_assert (sched_deps_info->generate_spec_deps + && !IS_SPECULATION_CHECK_P (insn)); + + if (HAS_INTERNAL_DEP (insn) + || SCHED_GROUP_P (insn)) + return -1; + + return sched_speculate_insn (insn, request, new_pat); } /* Print some information about block BB, which starts with HEAD and @@ -4281,70 +4119,6 @@ restore_bb_notes (basic_block first) bb_header = 0; } -/* Extend per basic block data structures of the scheduler. - If BB is NULL, initialize structures for the whole CFG. - Otherwise, initialize them for the just created BB. */ -void -sched_bbs_init (sbitmap blocks) -{ - rtx insn; - - gcc_assert (!sched_bbs || !blocks - || !sbitmap_any_common_bits (sched_bbs, blocks)); - - sched_last_basic_block = last_basic_block; - - if (sched_bbs) - sched_bbs = sbitmap_resize (sched_bbs, last_basic_block, 0); - else - { - sched_bbs = sbitmap_alloc (last_basic_block); - sbitmap_zero (sched_bbs); - } - - if (blocks) - sbitmap_a_or_b (sched_bbs, sched_bbs, blocks); - else - { - basic_block x; - - FOR_EACH_BB (x) - SET_BIT (sched_bbs, x->index); - } - - /* The following is done to keep current_sched_info->next_tail non null. */ - - insn = BB_END (EXIT_BLOCK_PTR->prev_bb); - if (NEXT_INSN (insn) == 0 - || (!NOTE_P (insn) - && !LABEL_P (insn) - /* Don't emit a NOTE if it would end up before a BARRIER. */ - && !BARRIER_P (NEXT_INSN (insn)))) - { - emit_note_after (NOTE_INSN_DELETED, insn); - /* Make insn to appear outside BB. */ - BB_END (EXIT_BLOCK_PTR->prev_bb) = insn; - } -} - -/* Add a basic block BB to extended basic block EBB. - If EBB is EXIT_BLOCK_PTR, then BB is recovery block. - If EBB is NULL, then BB should be a new region. */ -void -add_block (basic_block bb, basic_block ebb) -{ - gcc_assert (common_sched_info->detach_life_info - && bb->il.rtl->global_live_at_start == 0 - && bb->il.rtl->global_live_at_end == 0); - - glat_init (bb); - sched_bbs_init (NULL); - - if (common_sched_info->add_block) - /* This changes only data structures of the front-end. */ - common_sched_info->add_block (bb, ebb); -} - /* Helper function. Fix CFG after both in- and inter-block movement of control_flow_insn_p JUMP. */ @@ -4426,107 +4200,6 @@ move_succs (VEC(edge,gc) **succsp, basic_block to) *succsp = 0; } -/* Initialize GLAT (global_live_at_{start, end}) structures. - GLAT structures are used to substitute global_live_{start, end} - regsets during scheduling. This is necessary to use such functions as - split_block (), as they assume consistency of register live information. */ -static void -glat_init_1 (basic_block bb) -{ - glat_start[bb->index] = bb->il.rtl->global_live_at_start; - glat_end[bb->index] = bb->il.rtl->global_live_at_end; - - if (common_sched_info->detach_life_info) - { - bb->il.rtl->global_live_at_start = 0; - bb->il.rtl->global_live_at_end = 0; - } -} - -/* Initialize either for whole CFG or for just BB. */ -static void -glat_init (basic_block bb) -{ - if (common_sched_info->use_glat) - { - glat_start = xrealloc (glat_start, - last_basic_block * sizeof (*glat_start)); - glat_end = xrealloc (glat_end, last_basic_block * sizeof (*glat_end)); - - if (bb) - glat_init_1 (bb); - else - FOR_ALL_BB (bb) - glat_init_1 (bb); - } -} - -/* Attach reg_live_info back to basic blocks. - Also save regsets, that should not have been changed during scheduling, - for checking purposes (see check_reg_live). */ -void -attach_life_info (void) -{ - basic_block bb; - - FOR_ALL_BB (bb) - attach_life_info1 (bb); -} - -/* Helper function for attach_life_info. */ -static void -attach_life_info1 (basic_block bb) -{ - gcc_assert (bb->il.rtl->global_live_at_start == 0 - && bb->il.rtl->global_live_at_end == 0); - - if (glat_start[bb->index]) - { - gcc_assert (glat_end[bb->index]); - - bb->il.rtl->global_live_at_start = glat_start[bb->index]; - bb->il.rtl->global_live_at_end = glat_end[bb->index]; - - /* Make them NULL, so they won't be freed in free_glat. */ - glat_start[bb->index] = 0; - glat_end[bb->index] = 0; - -#ifdef ENABLE_CHECKING - if (bb->index < NUM_FIXED_BLOCKS - || common_sched_info->region_head_or_leaf_p (bb, 0)) - { - glat_start[bb->index] = ALLOC_REG_SET (®_obstack); - COPY_REG_SET (glat_start[bb->index], - bb->il.rtl->global_live_at_start); - } - - if (bb->index < NUM_FIXED_BLOCKS - || common_sched_info->region_head_or_leaf_p (bb, 1)) - { - glat_end[bb->index] = ALLOC_REG_SET (®_obstack); - COPY_REG_SET (glat_end[bb->index], bb->il.rtl->global_live_at_end); - } -#endif - } - else - { - gcc_assert (!glat_end[bb->index]); - - bb->il.rtl->global_live_at_start = ALLOC_REG_SET (®_obstack); - bb->il.rtl->global_live_at_end = ALLOC_REG_SET (®_obstack); - } -} - -/* Free per-basic block data. */ -void -sched_bbs_finish (void) -{ - sbitmap_free (sched_bbs); - sched_bbs = NULL; - - sched_last_basic_block = 0; -} - /* Remove INSN from the instruction stream. INSN should have any dependencies. */ static void @@ -4620,13 +4293,13 @@ debug_spec_status (ds_t s) FILE *f = stderr; if (s & BEGIN_DATA) - fprintf (f, "BEGIN_DATA: %d; ", get_dep_weak (s, BEGIN_DATA)); + fprintf (f, "BEGIN_DATA: %d; ", get_dep_weak_1 (s, BEGIN_DATA)); if (s & BE_IN_DATA) - fprintf (f, "BE_IN_DATA: %d; ", get_dep_weak (s, BE_IN_DATA)); + fprintf (f, "BE_IN_DATA: %d; ", get_dep_weak_1 (s, BE_IN_DATA)); if (s & BEGIN_CONTROL) - fprintf (f, "BEGIN_CONTROL: %d; ", get_dep_weak (s, BEGIN_CONTROL)); + fprintf (f, "BEGIN_CONTROL: %d; ", get_dep_weak_1 (s, BEGIN_CONTROL)); if (s & BE_IN_CONTROL) - fprintf (f, "BE_IN_CONTROL: %d; ", get_dep_weak (s, BE_IN_CONTROL)); + fprintf (f, "BE_IN_CONTROL: %d; ", get_dep_weak_1 (s, BE_IN_CONTROL)); if (s & HARD_DEP) fprintf (f, "HARD_DEP; "); @@ -4752,6 +4425,9 @@ check_cfg (rtx head, rtx tail) static void check_sched_flags (void) { + if (SEL_SCHED_P) + return; + if (flag_sched_stalled_insns) gcc_assert (!sched_deps_info->generate_spec_deps); if (sched_deps_info->generate_spec_deps) @@ -4763,6 +4439,274 @@ check_sched_flags (void) gcc_assert (common_sched_info->use_glat); } +#endif /* ENABLE_CHECKING */ + +const struct sched_scan_info_def *sched_scan_info; + +/* Extend per basic block data structures. */ +static void +extend_bb (void) +{ + if (sched_scan_info->extend_bb) + sched_scan_info->extend_bb (); +} + +/* Init data for BB. */ +static void +init_bb (basic_block bb) +{ + if (sched_scan_info->init_bb) + sched_scan_info->init_bb (bb); +} + +/* Extend per insn data structures. */ +static void +extend_insn (void) +{ + if (sched_scan_info->extend_insn) + sched_scan_info->extend_insn (); +} + +/* Init data structures for INSN. */ +static void +init_insn (rtx insn) +{ + if (sched_scan_info->init_insn) + sched_scan_info->init_insn (insn); +} + +/* Init all insns in BB. */ +static void +init_insns_in_bb (basic_block bb) +{ + rtx insn; + + FOR_BB_INSNS (bb, insn) + init_insn (insn); +} + +/* A driver function to add a set of basic blocks (BBS), + a single basic block (BB), a set of insns (INSNS) or a single insn (INSN) + to the scheduling region. + !!! This driver was only tested with one of the arguments non null. */ +void +sched_scan (const struct sched_scan_info_def *ssi, + bb_vec_t bbs, basic_block bb, insn_vec_t insns, rtx insn) +{ + sched_scan_info = ssi; + + if (bbs != NULL || bb != NULL) + { + extend_bb (); + + if (bbs != NULL) + { + unsigned i; + basic_block x; + + for (i = 0; VEC_iterate (basic_block, bbs, i, x); i++) + init_bb (x); + } + + if (bb != NULL) + init_bb (bb); + } + + extend_insn (); + + if (bbs != NULL) + { + unsigned i; + basic_block x; + + for (i = 0; VEC_iterate (basic_block, bbs, i, x); i++) + init_insns_in_bb (x); + } + + if (bb != NULL) + init_insns_in_bb (bb); + + if (insns != NULL) + { + unsigned i; + rtx x; + + for (i = 0; VEC_iterate (rtx, insns, i, x); i++) + init_insn (x); + } + + if (insn != NULL) + init_insn (insn); +} + +/* Pointers to GLAT data. See sched_inis_bb () for more information. */ +regset *glat_start = NULL, *glat_end = NULL; + +/* Extend per basic block data structures. */ +static void +sched_extend_bb (void) +{ + rtx insn; + + if (common_sched_info->use_glat) + { + glat_start = xrealloc (glat_start, + last_basic_block * sizeof (*glat_start)); + glat_end = xrealloc (glat_end, last_basic_block * sizeof (*glat_end)); + } + + /* The following is done to keep current_sched_info->next_tail non null. */ + insn = BB_END (EXIT_BLOCK_PTR->prev_bb); + if (NEXT_INSN (insn) == 0 + || (!NOTE_P (insn) + && !LABEL_P (insn) + /* Don't emit a NOTE if it would end up before a BARRIER. */ + && !BARRIER_P (NEXT_INSN (insn)))) + { + emit_note_after (NOTE_INSN_DELETED, insn); + /* Make insn to appear outside BB. */ + BB_END (EXIT_BLOCK_PTR->prev_bb) = insn; + } +} + +/* Init data structures for BB. */ +static void +sched_init_bb (basic_block bb) +{ + if (common_sched_info->use_glat) + { + /* Initialize GLAT (global_live_at_{start, end}) structures. + GLAT structures are used to substitute global_live_{start, end} + regsets during scheduling. This is necessary to use such functions as + split_block (), as they assume consistency of register live + information. */ + glat_start[bb->index] = bb->il.rtl->global_live_at_start; + glat_end[bb->index] = bb->il.rtl->global_live_at_end; + + if (common_sched_info->detach_life_info) + { + bb->il.rtl->global_live_at_start = NULL; + bb->il.rtl->global_live_at_end = NULL; + } + } +} + +/* Init per basic block data structures. */ +void +sched_init_bbs (bb_vec_t bbs, basic_block bb) +{ + const struct sched_scan_info_def ssi = + { + sched_extend_bb, /* extend_bb */ + sched_init_bb, /* init_bb */ + NULL, /* extend_insn */ + NULL /* init_insn */ + }; + + sched_scan (&ssi, bbs, bb, NULL, NULL); +} + +/* Finish per basic block data structures. */ +void +sched_finish_bbs (void) +{ + if (common_sched_info->use_glat) + { + /* Free GLAT information that was created by attach_life_info () + and used by sched-rgn.c: sched_rgn_global_finish (). */ +#ifdef ENABLE_CHECKING + if (common_sched_info->detach_life_info) + { + basic_block bb; + + FOR_ALL_BB (bb) + { + int i = bb->index; + + if (glat_start[i]) + FREE_REG_SET (glat_start[i]); + if (glat_end[i]) + FREE_REG_SET (glat_end[i]); + } + } +#endif + + free (glat_start); + glat_start = NULL; + + free (glat_end); + glat_end = NULL; + } +} + +/* Helper function for attach_life_info. */ +static void +attach_life_info1 (basic_block bb) +{ + int i = bb->index; + + gcc_assert (bb->il.rtl->global_live_at_start == NULL + && bb->il.rtl->global_live_at_end == NULL); + + if (glat_start[i] != NULL) + { + bb->il.rtl->global_live_at_start = glat_start[i]; + +#ifdef ENABLE_CHECKING + if (i < NUM_FIXED_BLOCKS + || common_sched_info->region_head_or_leaf_p (bb, 0)) + { + glat_start[i] = ALLOC_REG_SET (®_obstack); + COPY_REG_SET (glat_start[i], + bb->il.rtl->global_live_at_start); + } + else + /* Make it NULL, so it won't be freed in free_glat. */ + glat_start[i] = NULL; +#else + /* Make it NULL, so it won't be freed in free_glat. */ + glat_start[i] = NULL; +#endif + } + else + bb->il.rtl->global_live_at_start = ALLOC_REG_SET (®_obstack); + + if (glat_end[i] != NULL) + { + bb->il.rtl->global_live_at_end = glat_end[bb->index]; + +#ifdef ENABLE_CHECKING + if (bb->index < NUM_FIXED_BLOCKS + || common_sched_info->region_head_or_leaf_p (bb, 1)) + { + glat_end[bb->index] = ALLOC_REG_SET (®_obstack); + COPY_REG_SET (glat_end[bb->index], bb->il.rtl->global_live_at_end); + } + else + glat_end[i] = NULL; +#else + /* Make it NULL, so it won't be freed in free_glat. */ + glat_end[i] = NULL; +#endif + } + else + bb->il.rtl->global_live_at_end = ALLOC_REG_SET (®_obstack); +} + +/* Attach reg_live_info back to basic blocks. + Also save regsets, that should not have been changed during scheduling, + for checking purposes (see check_reg_live). */ +void +attach_life_info (void) +{ + basic_block bb; + + FOR_ALL_BB (bb) + attach_life_info1 (bb); +} + +#ifdef ENABLE_CHECKING + /* Check global_live_at_{start, end} regsets. If FATAL_P is TRUE, then abort execution at the first failure. Otherwise, print diagnostics to STDERR (this mode is for calling @@ -4803,245 +4747,154 @@ check_reg_live (bool fatal_p) } } } + #endif /* ENABLE_CHECKING */ +/* Mapping from instruction UID to its Logical UID. */ +VEC (int, heap) *sched_luids = NULL; + /* Extend data structures for logical insn UID. */ static void -sched_luid_extend (void) +luids_extend_insn (void) { - int sched_max_uid1 = get_max_uid () + 1; + int new_luids_max_uid = get_max_uid () + 1; - if (sched_insns) - sched_insns = sbitmap_resize (sched_insns, sched_max_uid1, 0); - else - { - sched_insns = sbitmap_alloc (sched_max_uid1); - sbitmap_zero (sched_insns); - } - - uid_to_luid = xrealloc (uid_to_luid, sched_max_uid1 * sizeof (*uid_to_luid)); - uid_to_luid[0] = 0; - - sched_max_uid = sched_max_uid1; + VEC_safe_grow_cleared (int, heap, sched_luids, new_luids_max_uid); } +/* Next LUID to assign to an instruction. */ +int sched_max_luid = 1; + /* Initialize LUID for INSN. */ static void -sched_luid_init (rtx insn) +luids_init_insn (rtx insn) { - int i = INSN_P (insn) ? 1 : common_sched_info->which_luid (insn); + int i = INSN_P (insn) ? 1 : common_sched_info->luid_for_non_insn (insn); + int luid; if (i >= 0) { - INSN_LUID (insn) = max_luid; + luid = sched_max_luid; - max_luid += i; + sched_max_luid += i; } else - INSN_LUID (insn) = -RAND_MAX; + luid = -1; + + SET_INSN_LUID (insn, luid); } -/* Free LUIDS. */ -static void -sched_luid_finish (void) +/* Initialize luids for BBS, BB, INSNS and INSN. + The hook common_sched_info->luid_for_non_insn () is used to determine + if notes, labels, etc. need luids. */ +void +sched_init_luids (bb_vec_t bbs, basic_block bb, insn_vec_t insns, rtx insn) { - free (uid_to_luid); - uid_to_luid = NULL; - - max_luid = 1; - - sbitmap_free (sched_insns); - sched_insns = NULL; + const struct sched_scan_info_def ssi = + { + NULL, /* extend_bb */ + NULL, /* init_bb */ + luids_extend_insn, /* extend_insn */ + luids_init_insn /* init_insn */ + }; - sched_max_uid = 0; + sched_scan (&ssi, bbs, bb, insns, insn); } -/* Extend per insn data structures. */ -static void -sched_insn_extend (void) +/* Free LUIDs. */ +void +sched_finish_luids (void) { - sched_deps_local_init (false); - - reg_weights = xrealloc (reg_weights, max_luid * sizeof (*reg_weights)); + VEC_free (int, heap, sched_luids); - if (common_sched_info->insn_extend) - common_sched_info->insn_extend (); + sched_max_luid = 1; +} +/* Extend per insn data in the target. */ +void +sched_extend_target (void) +{ if (targetm.sched.h_i_d_extended) targetm.sched.h_i_d_extended (); } -/* Init data structures for INSN. */ -static void -sched_insn_init (rtx insn) -{ - gcc_assert ((!BLOCK_FOR_INSN (insn) - || TEST_BIT (sched_bbs, BLOCK_NUM (insn))) - && !TEST_BIT (sched_insns, INSN_UID (insn))); - - if (INSN_LUID (insn) <= 0) - /* This is not an eligible insn. */ - return; - - find_insn_reg_weight (insn); - - SET_BIT (sched_insns, INSN_UID (insn)); - - if (common_sched_info->insn_init) - common_sched_info->insn_init (insn); -} +/* Haifa Instruction Data. */ +VEC (haifa_insn_data_def, heap) *h_i_d = NULL; -/* Free common per-insn data. */ +/* Extend global scheduler structures (those, that live across calls to + schedule_block) to include information about just emitted INSN. */ static void -sched_insn_finish (void) +extend_h_i_d (void) { - if (common_sched_info->insn_finish) - common_sched_info->insn_finish (); - - free (reg_weights); - reg_weights = NULL; + int new_h_i_d_size = get_max_uid () + 1; - sched_deps_local_finish (); + VEC_safe_grow_cleared (haifa_insn_data_def, heap, h_i_d, new_h_i_d_size); } -/* Extend per basic block data structures. */ +/* Initialize h_i_d entry of the INSN with default values. + Values, that are not explicitly initialized here, hold zero. */ static void -sched_bb_extend (void) +init_h_i_d (rtx insn) { - rtx insn; - - if (sched_bbs) - sched_bbs = sbitmap_resize (sched_bbs, last_basic_block, 0); - else + if (INSN_LUID (insn) > 0) { - sched_bbs = sbitmap_alloc (last_basic_block); - sbitmap_zero (sched_bbs); - } - - if (common_sched_info->use_glat) - { - glat_start = xrealloc (glat_start, - last_basic_block * sizeof (*glat_start)); - glat_end = xrealloc (glat_end, last_basic_block * sizeof (*glat_end)); + INSN_COST (insn) = -1; + find_insn_reg_weight (insn); + QUEUE_INDEX (insn) = QUEUE_NOWHERE; + INSN_TICK (insn) = INVALID_TICK; + INTER_TICK (insn) = INVALID_TICK; + TODO_SPEC (insn) = HARD_DEP; } +} - /* The following is done to keep current_sched_info->next_tail non null. */ - insn = BB_END (EXIT_BLOCK_PTR->prev_bb); - if (NEXT_INSN (insn) == 0 - || (!NOTE_P (insn) - && !LABEL_P (insn) - /* Don't emit a NOTE if it would end up before a BARRIER. */ - && !BARRIER_P (NEXT_INSN (insn)))) +/* Initialize haifa_insn_data for BBS, BB, INSNS and INSN. */ +void +haifa_init_h_i_d (bb_vec_t bbs, basic_block bb, insn_vec_t insns, rtx insn) +{ + const struct sched_scan_info_def ssi = { - emit_note_after (NOTE_INSN_DELETED, insn); - /* Make insn to appear outside BB. */ - BB_END (EXIT_BLOCK_PTR->prev_bb) = insn; - } - - if (common_sched_info->bb_extend) - common_sched_info->bb_extend (); + NULL, /* extend_bb */ + NULL, /* init_bb */ + extend_h_i_d, /* extend_insn */ + init_h_i_d /* init_insn */ + }; - /* Now, when sched frontend updated its state, we can finish extension of - per bb data structures. */ - sched_last_basic_block = last_basic_block; + sched_scan (&ssi, bbs, bb, insns, insn); } -/* Init data for BB. */ -static void -sched_bb_init (basic_block bb) +/* Finalize haifa_insn_data. */ +void +haifa_finish_h_i_d (void) { - rtx insn; - - gcc_assert (!TEST_BIT (sched_bbs, bb->index)); - - glat_init_1 (bb); - - SET_BIT (sched_bbs, bb->index); - - common_sched_info->remove_notes (bb); - - if (common_sched_info->bb_init) - common_sched_info->bb_init (bb); - - FOR_BB_INSNS (bb, insn) - if (!TEST_BIT (sched_insns, INSN_UID (insn))) - sched_insn_init (insn); + VEC_free (haifa_insn_data_def, heap, h_i_d); } -/* Free per-bb data. */ +/* Init data for the new insn INSN. */ static void -sched_bb_finish (void) +haifa_init_insn (rtx insn) { - if (common_sched_info->bb_finish) - common_sched_info->bb_finish (); - - /* NB: GLAT is freed in sched_finish (). */ + gcc_assert (insn != NULL); - sbitmap_free (sched_bbs); - sched_bbs = NULL; - - sched_last_basic_block = 0; + sched_init_luids (NULL, NULL, NULL, insn); + sched_extend_target (); + sched_deps_local_init (false); + haifa_init_h_i_d (NULL, NULL, NULL, insn); } -/* A driver function to add a set of basic blocks (BBS), - a single basic block (BB) or an INSN to the scheduling region. - Only one of the arguments can be non null. */ +/* Init data for the new basic block BB which comes after AFTER. */ void -sched_data_update (sbitmap bbs, basic_block bb, rtx insn) +haifa_init_only_bb (basic_block bb, basic_block after) { - sched_luid_extend (); - - if (insn) - { - gcc_assert (insn && INSN_NEED_LUID_P (insn)); - sched_luid_init (insn); + gcc_assert (bb != NULL + && common_sched_info->detach_life_info + && bb->il.rtl->global_live_at_start == NULL + && bb->il.rtl->global_live_at_end == NULL); - sched_insn_extend (); - sched_insn_init (insn); - } - else - { - sched_bb_extend (); - - if (bb) - { - FOR_BB_INSNS (bb, insn) - if (!TEST_BIT (sched_insns, INSN_UID (insn))) - sched_luid_init (insn); + sched_init_bbs (NULL, bb); - sched_insn_extend (); - - sched_bb_init (bb); - } - else - { - unsigned int i; - sbitmap_iterator sbi; - - EXECUTE_IF_SET_IN_SBITMAP (bbs, 0, i, sbi) - { - bb = BASIC_BLOCK (i); - - FOR_BB_INSNS (bb, insn) - if (!TEST_BIT (sched_insns, INSN_UID (insn))) - sched_luid_init (insn); - } - - sched_insn_extend (); - - EXECUTE_IF_SET_IN_SBITMAP (bbs, 0, i, sbi) - sched_bb_init (BASIC_BLOCK (i)); - } - } -} - -/* Free all common scheduling data. */ -void -sched_data_finish (void) -{ - sched_insn_finish (); - sched_bb_finish (); - sched_luid_finish (); + if (common_sched_info->add_block) + /* This changes only data structures of the front-end. */ + common_sched_info->add_block (bb, after); } #endif /* INSN_SCHEDULING */ diff --git a/gcc/modulo-sched.c b/gcc/modulo-sched.c index 4c67b9dc02b..12c805af858 100644 --- a/gcc/modulo-sched.c +++ b/gcc/modulo-sched.c @@ -248,14 +248,7 @@ compute_jump_reg_dependencies (rtx insn ATTRIBUTE_UNUSED, { } -static struct common_sched_info_def sms_common_sched_info = - { - NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, -#ifdef ENABLE_CHECKING - NULL, -#endif - 0, 0, SCHED_SMS_PASS - }; +static struct common_sched_info_def sms_common_sched_info; static struct sched_deps_info_def sms_sched_deps_info = { @@ -882,6 +875,19 @@ canon_loop (struct loop *loop) } } +/* Setup infos. */ +static void +setup_sched_infos (void) +{ + memcpy (&sms_common_sched_info, &haifa_common_sched_info, + sizeof (sms_common_sched_info)); + sms_common_sched_info.sched_pass_id = SCHED_SMS_PASS; + common_sched_info = &sms_common_sched_info; + + sched_deps_info = &sms_sched_deps_info; + current_sched_info = &sms_sched_info; +} + /* Probability in % that the sms-ed loop rolls enough so that optimized version may be entered. Just a guess. */ #define PROB_SMS_ENOUGH_ITERATIONS 80 @@ -923,9 +929,7 @@ sms_schedule (void) issue_rate = 1; /* Initialize the scheduler. */ - common_sched_info = &sms_common_sched_info; - sched_deps_info = &sms_sched_deps_info; - current_sched_info = &sms_sched_info; + setup_sched_infos (); haifa_sched_init (); diff --git a/gcc/params.def b/gcc/params.def index 65fa352f993..ce1d914cc2a 100644 --- a/gcc/params.def +++ b/gcc/params.def @@ -552,6 +552,11 @@ DEFPARAM(PARAM_SELSCHED_MAX_LOOKAHEAD, "The maximum size of the lookahead window of selective scheduling", 32, 0, 0) +DEFPARAM(PARAM_SELSCHED_MAX_SCHED_TIMES, + "selsched-max-spec-ready", + "The maximum spec for an insn to be considered ready", + 2, 0, 0) + /* Minimal distance (in CPU cycles) between store and load targeting same memory locations. */ @@ -560,18 +565,33 @@ DEFPARAM (PARAM_SCHED_MEM_TRUE_DEP_COST, "Minimal distance between possibly conflicting store and load", 1, 0, 0) -DEFPARAM(PARAM_ALLOW_START, - "allow_start", +DEFPARAM(PARAM_SEL1_START, + "sel1-start", + "Allow something", + 0, 0, 0) + +DEFPARAM(PARAM_SEL1_STOP, + "sel1-stop", + "Allow something", + 0, 0, 0) + +DEFPARAM(PARAM_SEL1_P, + "sel1-p", + "Allow something", + 0, 0, 0) + +DEFPARAM(PARAM_SEL2_START, + "sel2-start", "Allow something", 0, 0, 0) -DEFPARAM(PARAM_ALLOW_STOP, - "allow_stop", +DEFPARAM(PARAM_SEL2_STOP, + "sel2-stop", "Allow something", 0, 0, 0) -DEFPARAM(PARAM_ALLOW_P, - "allow_p", +DEFPARAM(PARAM_SEL2_P, + "sel2-p", "Allow something", 0, 0, 0) diff --git a/gcc/rtlanal.c b/gcc/rtlanal.c index 2d156638b6d..1eaaef4f462 100644 --- a/gcc/rtlanal.c +++ b/gcc/rtlanal.c @@ -2090,6 +2090,13 @@ may_trap_p_1 (rtx x, unsigned flags) if (x == 0) return 0; code = GET_CODE (x); + + if (code == UNSPEC + && (targetm.sched.skip_rtx_p == NULL + || !targetm.sched.skip_rtx_p (x))) + /* Support ia64 speculation. */ + return may_trap_p_1 (XVECEXP (x, 0, 0), flags); + switch (code) { /* Handle these cases quickly. */ diff --git a/gcc/sched-deps.c b/gcc/sched-deps.c index e5364d27d11..372ac0203a2 100644 --- a/gcc/sched-deps.c +++ b/gcc/sched-deps.c @@ -48,10 +48,6 @@ Software Foundation, 51 Franklin Street, Fifth Floor, Boston, MA /* Holds current parameters for the dependency analyzer. */ struct sched_deps_info_def *sched_deps_info; -/* An array indexed by INSN_UID that holds the data related - to insn's dependencies and common to all schedulers. */ -struct deps_insn_data *d_i_d = NULL; - /* Same as above, but the data is specific to the Haifa scheduler. */ struct haifa_deps_insn_data *h_d_i_d = NULL; @@ -735,6 +731,7 @@ static void haifa_start_insn (rtx insn) { gcc_assert (insn && !cur_insn); + cur_insn = insn; } @@ -784,12 +781,40 @@ haifa_note_dep (rtx elem, ds_t ds) NULL); } -#define note_reg_use(R) (sched_deps_info->note_reg_use (R)) -#define note_reg_set(R) (sched_deps_info->note_reg_set (R)) -#define note_reg_clobber(R) (sched_deps_info->note_reg_clobber (R)) -#define note_mem_dep(M1, M2, E, DS) \ -(sched_deps_info->note_mem_dep (M1,M2,E,DS)) -#define note_dep(E, DS) (sched_deps_info->note_dep (E, DS)) +static void +note_reg_use (int r) +{ + if (sched_deps_info->note_reg_use) + sched_deps_info->note_reg_use (r); +} + +static void +note_reg_set (int r) +{ + if (sched_deps_info->note_reg_set) + sched_deps_info->note_reg_set (r); +} + +static void +note_reg_clobber (int r) +{ + if (sched_deps_info->note_reg_clobber) + sched_deps_info->note_reg_clobber (r); +} + +static void +note_mem_dep (rtx m1, rtx m2, rtx e, ds_t ds) +{ + if (sched_deps_info->note_mem_dep) + sched_deps_info->note_mem_dep (m1,m2,e,ds); +} + +static void +note_dep (rtx e, ds_t ds) +{ + if (sched_deps_info->note_dep) + sched_deps_info->note_dep (e, ds); +} /* Return corresponding to DS reg_note. */ enum reg_note @@ -1212,8 +1237,23 @@ sched_analyze_2 (struct deps *deps, rtx x, rtx insn) } for (u = deps->last_pending_memory_flush; u; u = XEXP (u, 1)) - if (! JUMP_P (XEXP (u, 0)) || deps_may_trap_p (x)) - add_dependence (insn, XEXP (u, 0), REG_DEP_ANTI); + { + if (! JUMP_P (XEXP (u, 0))) + add_dependence (insn, XEXP (u, 0), REG_DEP_ANTI); + else if (deps_may_trap_p (x)) + { + if ((sched_deps_info->generate_spec_deps) + && SEL_SCHED_P && (spec_info->mask & BEGIN_CONTROL)) + { + ds_t ds = set_dep_weak (DEP_ANTI, BEGIN_CONTROL, + MAX_DEP_WEAK); + + note_dep (XEXP (u, 0), ds); + } + else + add_dependence (insn, XEXP (u, 0), REG_DEP_ANTI); + } + } /* Always add these dependencies to pending_reads, since this insn may be followed by a write. */ @@ -1695,9 +1735,6 @@ deps_analyze_insn (struct deps *deps, rtx insn) if (NONJUMP_INSN_P (insn) || JUMP_P (insn)) { - /* Clear out the stale LOG_LINKS from flow. */ - free_INSN_LIST_list (&LOG_LINKS (insn)); - /* Make each JUMP_INSN a scheduling barrier for memory references. */ if (JUMP_P (insn)) @@ -1718,9 +1755,6 @@ deps_analyze_insn (struct deps *deps, rtx insn) CANT_MOVE (insn) = 1; - /* Clear out the stale LOG_LINKS from flow. */ - free_INSN_LIST_list (&LOG_LINKS (insn)); - if (find_reg_note (insn, REG_SETJMP, NULL)) { /* This is setjmp. Assume that all registers, not just @@ -1879,6 +1913,11 @@ sched_analyze (struct deps *deps, rtx head, rtx tail) for (insn = head;; insn = NEXT_INSN (insn)) { + + if (INSN_P (insn)) + /* Clear out the stale LOG_LINKS from flow. */ + free_INSN_LIST_list (&LOG_LINKS (insn)); + deps_analyze_insn (deps, insn); if (insn == tail) @@ -2031,18 +2070,31 @@ free_deps (struct deps *deps) free (deps->reg_last); } +/* An array indexed by INSN_UID that holds the data related + to insn's dependencies and common to all schedulers. */ +VEC (deps_insn_data_def, heap) *d_i_d = NULL; + +void +deps_extend_d_i_d (void) +{ + VEC_safe_grow_cleared (deps_insn_data_def, heap, d_i_d, sched_max_luid); +} + +void +deps_finish_d_i_d (void) +{ + VEC_free (deps_insn_data_def, heap, d_i_d); +} + /* If it is profitable to use them, initialize or extend (depending on CREATE_P) caches for tracking dependency information. */ void sched_deps_local_init (bool create_caches_p) { - int i; + deps_extend_d_i_d (); - d_i_d = xrecalloc (d_i_d, max_luid, cur_max_luid, sizeof (*d_i_d)); - for (i = cur_max_luid; i < max_luid; i++) - d_i_d[i].cost = -1; - - h_d_i_d = xrecalloc (h_d_i_d, max_luid, cur_max_luid, sizeof (*h_d_i_d)); + h_d_i_d = xrecalloc (h_d_i_d, sched_max_luid, cur_max_luid, + sizeof (*h_d_i_d)); /* FIXME: We need another caching mechanism for selective scheduling, so we don't use this one. */ @@ -2055,28 +2107,30 @@ sched_deps_local_init (bool create_caches_p) the comment before the declaration of true_dependency_cache for what we consider "very high". */ if (true_dependency_cache - || (create_caches_p && (max_luid / n_basic_blocks > 100 * 5))) + || (create_caches_p && (sched_max_luid / n_basic_blocks > 100 * 5))) { int i; true_dependency_cache = XRESIZEVEC (bitmap_head, - true_dependency_cache, max_luid); + true_dependency_cache, + sched_max_luid); output_dependency_cache = XRESIZEVEC (bitmap_head, output_dependency_cache, - max_luid); + sched_max_luid); anti_dependency_cache = XRESIZEVEC (bitmap_head, - anti_dependency_cache, max_luid); + anti_dependency_cache, + sched_max_luid); #ifdef ENABLE_CHECKING forward_dependency_cache = XRESIZEVEC (bitmap_head, forward_dependency_cache, - max_luid); + sched_max_luid); #endif if (sched_deps_info->generate_spec_deps) spec_dependency_cache = XRESIZEVEC (bitmap_head, spec_dependency_cache, - max_luid); + sched_max_luid); - for (i = cur_max_luid; i < max_luid; i++) + for (i = cur_max_luid; i < sched_max_luid; i++) { bitmap_initialize (&true_dependency_cache[i], 0); bitmap_initialize (&output_dependency_cache[i], 0); @@ -2090,7 +2144,7 @@ sched_deps_local_init (bool create_caches_p) } } - cur_max_luid = max_luid; + cur_max_luid = sched_max_luid; } /* Free the caches allocated in init_dependency_caches. */ @@ -2130,12 +2184,12 @@ sched_deps_local_finish (void) } - free (d_i_d); - d_i_d = NULL; free (h_d_i_d); h_d_i_d = NULL; cur_max_luid = 0; + + deps_finish_d_i_d (); } /* Initialize some global variables needed by the dependency analysis @@ -2392,9 +2446,10 @@ delete_back_forw_dep (rtx insn, rtx elem) /* Return weakness of speculative type TYPE in the dep_status DS. */ dw_t -get_dep_weak (ds_t ds, ds_t type) +get_dep_weak_1 (ds_t ds, ds_t type) { ds = ds & type; + switch (type) { case BEGIN_DATA: ds >>= BEGIN_DATA_BITS_OFFSET; break; @@ -2404,10 +2459,18 @@ get_dep_weak (ds_t ds, ds_t type) default: gcc_unreachable (); } - gcc_assert (MIN_DEP_WEAK <= ds && ds <= MAX_DEP_WEAK); return (dw_t) ds; } +dw_t +get_dep_weak (ds_t ds, ds_t type) +{ + dw_t dw = get_dep_weak_1 (ds, type); + + gcc_assert (MIN_DEP_WEAK <= dw && dw <= MAX_DEP_WEAK); + return dw; +} + /* Return the dep_status, which has the same parameters as DS, except for speculative type TYPE, that will have weakness DW. */ ds_t @@ -2427,9 +2490,12 @@ set_dep_weak (ds_t ds, ds_t type, dw_t dw) return ds; } -/* Return the join of two dep_statuses DS1 and DS2. */ -ds_t -ds_merge (ds_t ds1, ds_t ds2) +/* Return the join of two dep_statuses DS1 and DS2. + If MAX_P is true then choose the greater probability, + otherwise multiply probabilities. + This function assumes that both DS1 and DS2 contain speculative bits. */ +static ds_t +ds_merge_1 (ds_t ds1, ds_t ds2, bool max_p) { ds_t ds, t; @@ -2446,12 +2512,24 @@ ds_merge (ds_t ds1, ds_t ds2) ds |= ds2 & t; else if ((ds1 & t) && (ds2 & t)) { + dw_t dw1 = get_dep_weak (ds1, t); + dw_t dw2 = get_dep_weak (ds2, t); ds_t dw; - dw = ((ds_t) get_dep_weak (ds1, t)) * ((ds_t) get_dep_weak (ds2, t)); - dw /= MAX_DEP_WEAK; - if (dw < MIN_DEP_WEAK) - dw = MIN_DEP_WEAK; + if (!max_p) + { + dw = ((ds_t) dw1) * ((ds_t) dw2); + dw /= MAX_DEP_WEAK; + if (dw < MIN_DEP_WEAK) + dw = MIN_DEP_WEAK; + } + else + { + if (dw1 >= dw2) + dw = dw1; + else + dw = dw2; + } ds = set_dep_weak (ds, t, (dw_t) dw); } @@ -2465,6 +2543,128 @@ ds_merge (ds_t ds1, ds_t ds2) return ds; } +/* Return the join of two dep_statuses DS1 and DS2. + This function assumes that both DS1 and DS2 contain speculative bits. */ +ds_t +ds_merge (ds_t ds1, ds_t ds2) +{ + return ds_merge_1 (ds1, ds2, false); +} + +/* Return the join of two dep_statuses DS1 and DS2. */ +ds_t +ds_full_merge (ds_t ds, ds_t ds2, rtx mem1, rtx mem2) +{ + ds_t new_status = ds | ds2; + + if (new_status & SPECULATIVE) + { + if ((ds && !(ds & SPECULATIVE)) + || (ds2 && !(ds2 & SPECULATIVE))) + /* Then this dep can't be speculative. */ + new_status &= ~SPECULATIVE; + else + { + /* Both are speculative. Merging probabilities. */ + if (mem1) + { + dw_t dw; + + dw = estimate_dep_weak (mem1, mem2); + ds = set_dep_weak (ds, BEGIN_DATA, dw); + } + + if (!ds) + new_status = ds2; + else if (!ds2) + new_status = ds; + else + new_status = ds_merge (ds2, ds); + } + } + + return new_status; +} + +/* Return the join of DS1 and DS2. Use maximum instead of multiplying + probabilities. */ +ds_t +ds_max_merge (ds_t ds1, ds_t ds2) +{ + if (ds1 == 0 && ds2 == 0) + return 0; + + return ds_merge_1 (ds1, ds2, true); +} + +/* Return the probability of speculation success for the speculation + status DS. */ +dw_t +ds_weak (ds_t ds) +{ + ds_t res = 1, dt; + int n = 0; + + dt = FIRST_SPEC_TYPE; + do + { + if (ds & dt) + { + res *= (ds_t) get_dep_weak (ds, dt); + n++; + } + + if (dt == LAST_SPEC_TYPE) + break; + dt <<= SPEC_TYPE_SHIFT; + } + while (1); + + gcc_assert (n); + while (--n) + res /= MAX_DEP_WEAK; + + if (res < MIN_DEP_WEAK) + res = MIN_DEP_WEAK; + + gcc_assert (res <= MAX_DEP_WEAK); + + return (dw_t) res; +} + +/* Return a dep status that contains all speculation types of DS. */ +ds_t +ds_get_speculation_types (ds_t ds) +{ + if (ds & BEGIN_DATA) + ds |= BEGIN_DATA; + if (ds & BE_IN_DATA) + ds |= BE_IN_DATA; + if (ds & BEGIN_CONTROL) + ds |= BEGIN_CONTROL; + if (ds & BE_IN_CONTROL) + ds |= BE_IN_CONTROL; + + return ds & SPECULATIVE; +} + +/* Return a dep status that contains maximal weakness for each speculation + type present in DS. */ +ds_t +ds_get_max_dep_weak (ds_t ds) +{ + if (ds & BEGIN_DATA) + ds = set_dep_weak (ds, BEGIN_DATA, MAX_DEP_WEAK); + if (ds & BE_IN_DATA) + ds = set_dep_weak (ds, BE_IN_DATA, MAX_DEP_WEAK); + if (ds & BEGIN_CONTROL) + ds = set_dep_weak (ds, BEGIN_CONTROL, MAX_DEP_WEAK); + if (ds & BE_IN_CONTROL) + ds = set_dep_weak (ds, BE_IN_CONTROL, MAX_DEP_WEAK); + + return ds; +} + #ifdef INSN_SCHEDULING #ifdef ENABLE_CHECKING /* Verify that dependence type and status are consistent. diff --git a/gcc/sched-deps.h b/gcc/sched-deps.h index d8b3e619cc5..0275fb90ba2 100644 --- a/gcc/sched-deps.h +++ b/gcc/sched-deps.h @@ -22,6 +22,8 @@ Software Foundation, 51 Franklin Street, Fifth Floor, Boston, MA #ifndef GCC_SCHED_DEPS_H #define GCC_SCHED_DEPS_H +#include "sched-int.h" + struct haifa_deps_insn_data { /* The number of incoming edges in the forward dependency graph. @@ -32,44 +34,38 @@ struct haifa_deps_insn_data /* Nonzero if instruction has internal dependence (e.g. add_dependence was invoked with (insn == elem)). */ unsigned int has_internal_dep; -}; -struct deps_insn_data -{ /* A list of insns which depend on the instruction. Unlike LOG_LINKS, it represents forward dependencies. Note that we want to use this in selective scheduling also, so it has moved from h_d_i_d. */ rtx depend; +}; - /* A priority for each insn. */ - int priority; - - /* A cost of this instruction. */ - short cost; +extern struct haifa_deps_insn_data *h_d_i_d; - /* Nonzero if priority has been computed already. */ - unsigned int priority_known : 1; +#define HDID(INSN) (h_d_i_d[INSN_LUID (INSN)]) +#define INSN_DEP_COUNT(INSN) (HDID (INSN).dep_count) +#define HAS_INTERNAL_DEP(INSN) (HDID (INSN).has_internal_dep) +#define INSN_DEPEND(INSN) (HDID (INSN).depend) +struct _deps_insn_data +{ /* Some insns (e.g. call) are not allowed to move across blocks. */ unsigned int cant_move : 1; - - /* Mark insns that may trap so we don't move them through jumps. */ - unsigned int may_trap : 1; }; -extern struct haifa_deps_insn_data *h_d_i_d; -extern struct deps_insn_data *d_i_d; +typedef struct _deps_insn_data deps_insn_data_def; +typedef deps_insn_data_def *deps_insn_data_t; -#define DID(INSN) (d_i_d[INSN_LUID (INSN)]) -#define HDID(INSN) (h_d_i_d[INSN_LUID (INSN)]) +DEF_VEC_O (deps_insn_data_def); +DEF_VEC_ALLOC_O (deps_insn_data_def, heap); -#define INSN_DEPEND(INSN) (DID (INSN).depend) -#define INSN_DEP_COUNT(INSN) (HDID (INSN).dep_count) -#define HAS_INTERNAL_DEP(INSN) (HDID (INSN).has_internal_dep) -#define CANT_MOVE(INSN) (DID (INSN).cant_move) -#define MAY_TRAP(INSN) (DID (INSN).may_trap) +extern VEC (deps_insn_data_def, heap) *d_i_d; + +#define DID(INSN) (VEC_index (deps_insn_data_def, d_i_d, INSN_LUID (INSN))) +#define CANT_MOVE(INSN) (DID (INSN)->cant_move) struct sched_deps_info_def { @@ -151,10 +147,19 @@ extern enum DEPS_ADJUST_RESULT add_or_update_back_dep (rtx, rtx, extern void add_or_update_back_forw_dep (rtx, rtx, enum reg_note, ds_t); extern void add_back_forw_dep (rtx, rtx, enum reg_note, ds_t); extern void delete_back_forw_dep (rtx, rtx); +extern dw_t get_dep_weak_1 (ds_t, ds_t); extern dw_t get_dep_weak (ds_t, ds_t); extern ds_t set_dep_weak (ds_t, ds_t, dw_t); extern dw_t estimate_dep_weak (rtx, rtx); extern ds_t ds_merge (ds_t, ds_t); +extern ds_t ds_full_merge (ds_t, ds_t, rtx, rtx); +extern ds_t ds_max_merge (ds_t, ds_t); +extern dw_t ds_weak (ds_t); +extern ds_t ds_get_speculation_types (ds_t); +extern ds_t ds_get_max_dep_weak (ds_t); + +extern void deps_extend_d_i_d (void); +extern void deps_finish_d_i_d (void); extern void sched_deps_local_init (bool); extern void sched_deps_local_finish (void); diff --git a/gcc/sched-ebb.c b/gcc/sched-ebb.c index ff233182f26..f19dd651ec2 100644 --- a/gcc/sched-ebb.c +++ b/gcc/sched-ebb.c @@ -44,11 +44,12 @@ Software Foundation, 51 Franklin Street, Fifth Floor, Boston, MA #include "target.h" #include "output.h" -/* The number of insns scheduled so far. */ -static int sched_n_insns; /* The number of insns to be scheduled in total. */ -static int n_insns; +static int rgn_n_insns; + +/* The number of insns scheduled so far. */ +static int sched_rgn_n_insns; /* Set of blocks, that already have their dependencies calculated. */ static bitmap_head dont_calc_deps; @@ -70,13 +71,13 @@ static void add_deps_for_risky_insns (rtx, rtx); static basic_block schedule_ebb (rtx, rtx); static void debug_ebb_dependencies (rtx, rtx); -static void add_remove_insn (rtx, int); -static void add_block1 (basic_block, basic_block); +static void ebb_add_remove_insn (rtx, int); +static void ebb_add_block (basic_block, basic_block); static basic_block advance_target_bb (basic_block, rtx); -static void fix_recovery_cfg (int, int, int); +static void ebb_fix_recovery_cfg (int, int, int); #ifdef ENABLE_CHECKING -static int ebb_head_or_leaf_p (basic_block, int); +static int ebb_region_head_or_leaf_p (basic_block, int); #endif /* Return nonzero if there are more insns that should be scheduled. */ @@ -84,7 +85,7 @@ static int ebb_head_or_leaf_p (basic_block, int); static int schedule_more_p (void) { - return sched_n_insns < n_insns; + return sched_rgn_n_insns < rgn_n_insns; } /* Debug dependencies between HEAD and TAIL. @@ -172,7 +173,7 @@ init_ready_list (void) rtx next_tail = current_sched_info->next_tail; rtx insn; - sched_n_insns = 0; + sched_rgn_n_insns = 0; #if 1 /* Print debugging information. */ @@ -188,14 +189,14 @@ init_ready_list (void) n++; } - gcc_assert (n == n_insns); + gcc_assert (n == rgn_n_insns); } /* INSN is being scheduled after LAST. Update counters. */ static void begin_schedule_ready (rtx insn, rtx last) { - sched_n_insns++; + sched_rgn_n_insns++; if (BLOCK_FOR_INSN (insn) == last_bb /* INSN is a jump in the last block, ... */ @@ -251,7 +252,8 @@ begin_schedule_ready (rtx insn, rtx last) current_sched_info->next_tail = NEXT_INSN (BB_END (bb)); gcc_assert (current_sched_info->next_tail); - add_block (bb, last_bb); + /* Append new basic block to the end of the ebb. */ + haifa_init_only_bb (bb, last_bb); gcc_assert (last_bb == bb); } } @@ -333,15 +335,7 @@ ebb_compute_jump_reg_dependencies (rtx insn, regset cond_set, regset used, /* Used in schedule_insns to initialize current_sched_info for scheduling regions (or single basic blocks). */ -static struct common_sched_info_def ebb_common_sched_info = - { - fix_recovery_cfg, add_block1, - NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, -#ifdef ENABLE_CHECKING - ebb_head_or_leaf_p, -#endif - 1, 1, SCHED_EBB_PASS - }; +static struct common_sched_info_def ebb_common_sched_info; static struct sched_deps_info_def ebb_sched_deps_info = { @@ -365,9 +359,8 @@ static struct haifa_sched_info ebb_sched_info = NULL, NULL, 1, 0, - add_remove_insn, + ebb_add_remove_insn, begin_schedule_ready, - /*add_block1,*/ advance_target_bb }; @@ -563,7 +556,7 @@ schedule_ebb (rtx head, rtx tail) /* Set priorities. */ current_sched_info->sched_max_insns_priority = 0; - n_insns = set_priorities (head, tail); + rgn_n_insns = set_priorities (head, tail); current_sched_info->sched_max_insns_priority++; current_sched_info->prev_head = PREV_INSN (head); @@ -575,15 +568,17 @@ schedule_ebb (rtx head, rtx tail) target_bb = first_bb; - haifa_local_init (n_insns); + /* Make ready list big enough to hold all the instructions from the ebb. */ + sched_extend_ready_list (rgn_n_insns); schedule_block (&target_bb); - sched_local_finish (); + /* Free ready list. */ + sched_finish_ready_list (); /* We might pack all instructions into fewer blocks, so we may made some of them empty. Can't assert (b == last_bb). */ /* Sanity check: verify that all region insns were scheduled. */ - gcc_assert (sched_n_insns == n_insns); + gcc_assert (sched_rgn_n_insns == rgn_n_insns); if (EDGE_COUNT (last_bb->preds) == 0) /* LAST_BB is unreachable. */ @@ -619,9 +614,24 @@ schedule_ebbs (void) if (n_basic_blocks == NUM_FIXED_BLOCKS) return; - common_sched_info = &ebb_common_sched_info; - sched_deps_info = &ebb_sched_deps_info; - current_sched_info = &ebb_sched_info; + /* Setup infos. */ + { + memcpy (&ebb_common_sched_info, &haifa_common_sched_info, + sizeof (ebb_common_sched_info)); + + ebb_common_sched_info.fix_recovery_cfg = ebb_fix_recovery_cfg; + ebb_common_sched_info.add_block = ebb_add_block; + ebb_common_sched_info.region_head_or_leaf_p = ebb_region_head_or_leaf_p; + ebb_common_sched_info.use_glat = 1; + ebb_common_sched_info.detach_life_info = 1; + ebb_common_sched_info.sched_pass_id = SCHED_EBB_PASS; + + common_sched_info = &ebb_common_sched_info; + + sched_deps_info = &ebb_sched_deps_info; + + current_sched_info = &ebb_sched_info; + } haifa_sched_init (); @@ -748,17 +758,17 @@ schedule_ebbs (void) /* INSN has been added to/removed from current ebb. */ static void -add_remove_insn (rtx insn ATTRIBUTE_UNUSED, int remove_p) +ebb_add_remove_insn (rtx insn ATTRIBUTE_UNUSED, int remove_p) { if (!remove_p) - n_insns++; + rgn_n_insns++; else - n_insns--; + rgn_n_insns--; } /* BB was added to ebb after AFTER. */ static void -add_block1 (basic_block bb, basic_block after) +ebb_add_block (basic_block bb, basic_block after) { /* Recovery blocks are always bounded by BARRIERS, therefore, they always form single block EBB, @@ -811,7 +821,8 @@ advance_target_bb (basic_block bb, rtx insn) For parameter meaning please refer to sched-int.h: struct sched_info: fix_recovery_cfg. */ static void -fix_recovery_cfg (int bbi ATTRIBUTE_UNUSED, int jump_bbi, int jump_bb_nexti) +ebb_fix_recovery_cfg (int bbi ATTRIBUTE_UNUSED, int jump_bbi, + int jump_bb_nexti) { gcc_assert (last_bb->index != bbi); @@ -824,7 +835,7 @@ fix_recovery_cfg (int bbi ATTRIBUTE_UNUSED, int jump_bbi, int jump_bb_nexti) current ebb. For more information please refer to sched-int.h: struct sched_info: region_head_or_leaf_p. */ static int -ebb_head_or_leaf_p (basic_block bb, int leaf_p) +ebb_region_head_or_leaf_p (basic_block bb, int leaf_p) { if (!leaf_p) return bitmap_bit_p (&ebb_head, bb->index); diff --git a/gcc/sched-int.h b/gcc/sched-int.h index 940b7118019..521ebe3971f 100644 --- a/gcc/sched-int.h +++ b/gcc/sched-int.h @@ -28,12 +28,57 @@ Software Foundation, 51 Franklin Street, Fifth Floor, Boston, MA /* For state_t. */ #include "insn-attr.h" +/* For VEC (int, heap). */ +#include "vecprim.h" + extern int sched_verbose_param; /* Identificator of a scheduler pass. */ enum sched_pass_id_t { SCHED_PASS_UNKNOWN, SCHED_RGN_PASS, SCHED_EBB_PASS, SCHED_SMS_PASS, SCHED_SEL_PASS }; +typedef VEC (basic_block, heap) *bb_vec_t; +typedef VEC (rtx, heap) *insn_vec_t; + +struct sched_scan_info_def +{ + /* This hook notifies scheduler frontend to extend its internal per basic + block data structures. This hook should be called once before a series of + calls to bb_init (). */ + void (*extend_bb) (void); + + /* This hook makes scheduler frontend to initialize its internal data + structures for the passed basic block. */ + void (*init_bb) (basic_block); + + /* This hook notifies scheduler frontend to extend its internal per insn data + structures. This hook should be called once before a series of calls to + insn_init (). */ + void (*extend_insn) (void); + + /* This hook makes scheduler frontend to initialize its internal data + structures for the passed insn. */ + void (*init_insn) (rtx); +}; + +extern const struct sched_scan_info_def *sched_scan_info; + +extern void sched_scan (const struct sched_scan_info_def *, + bb_vec_t, basic_block, insn_vec_t, rtx); + +extern void sched_init_bbs (bb_vec_t, basic_block); +extern void sched_finish_bbs (void); + +extern void sched_init_luids (bb_vec_t, basic_block, insn_vec_t, rtx); +extern void sched_finish_luids (void); + +extern void sched_extend_target (void); + +extern void haifa_init_h_i_d (bb_vec_t, basic_block, insn_vec_t, rtx); +extern void haifa_finish_h_i_d (void); + +extern void haifa_init_only_bb (basic_block, basic_block); + /* Hooks that are common to all the schedulers. */ struct common_sched_info_def { @@ -59,36 +104,7 @@ struct common_sched_info_def -1 - if this rtx don't need a luid. 0 - if it should have the same luid as the previous insn. 1 - if it needs a separate luid. */ - int (*which_luid) (rtx); - - /* Remove unneccesary notes from the given basic block. */ - void (*remove_notes) (basic_block); - - /* This hook notifies scheduler frontend to extend its internal per basic - block data structures. This hook should be called once before a series of - calls to bb_init (). */ - void (*bb_extend) (void); - - /* This hook makes scheduler frontend to initialize its internal data - structures for the passed basic block. */ - void (*bb_init) (basic_block); - - /* This hook notifies scheduler frontend that it should finalize its internal - per basic block data structures. */ - void (*bb_finish) (void); - - /* This hook notifies scheduler frontend to extend its internal per insn data - structures. This hook should be called once before a series of calls to - insn_init (). */ - void (*insn_extend) (void); - - /* This hook makes scheduler frontend to initialize its internal data - structures for the passed insn. */ - void (*insn_init) (rtx); - - /* This hook notifies scheduler frontend that it should finalize its internal - per insn data structures. */ - void (*insn_finish) (void); + int (*luid_for_non_insn) (rtx); #ifdef ENABLE_CHECKING /* If the second parameter is zero, return nonzero, if block is head of the @@ -116,10 +132,7 @@ struct common_sched_info_def extern struct common_sched_info_def *common_sched_info; -/* Return true if INSN should have a separate luid. */ -#define INSN_NEED_LUID_P(INSN) (INSN_P (INSN) \ -|| ((NOTE_P (INSN) || LABEL_P (INSN)) \ -&& common_sched_info->which_luid (INSN) == 1)) +extern const struct common_sched_info_def haifa_common_sched_info; /* Return true if selective scheduling pass is working. */ #define SEL_SCHED_P (common_sched_info->sched_pass_id == SCHED_SEL_PASS) @@ -133,26 +146,15 @@ extern int sched_emulate_haifa_p; for each basic block. */ extern regset *glat_start, *glat_end; -/* The highest INSN_UID scheduler knows of. All insns with uids greater than - this don't have their scheduler data structures initialized. */ -extern int sched_max_uid; - -/* The highest basic block index scheduler knows of. All basic block with - indecies greater than this don't have their scheduler data structures - initialized. */ -extern int sched_last_basic_block; - /* Mapping from INSN_UID to INSN_LUID. In the end all other per insn data structures should be indexed by luid. */ -extern int *uid_to_luid; -#define INSN_LUID(INSN) (uid_to_luid[INSN_UID (INSN)]) +extern VEC (int, heap) *sched_luids; +#define INSN_LUID(INSN) (VEC_index (int, sched_luids, INSN_UID (INSN))) +#define SET_INSN_LUID(INSN, LUID) \ +(VEC_replace (int, sched_luids, INSN_UID (INSN), (LUID))) /* The highest INSN_LUID. */ -extern int max_luid; - -/* An array to hold register weights for each insn. Indexed by luid. */ -extern short *reg_weights; -#define INSN_REG_WEIGHT(INSN) (reg_weights[INSN_LUID (INSN)]) +extern int sched_max_luid; /* Return true if NOTE is a note but not a basic block one. */ #define NOTE_NOT_BB_P(NOTE) (NOTE_P (NOTE) && (NOTE_LINE_NUMBER (NOTE) \ @@ -166,15 +168,7 @@ extern int sched_verbose; finished. */ extern rtx note_list; -/* A set of basic blocks that are in current scheduling region. */ -extern sbitmap sched_blocks; - -/* A set of instructions that are in current scheduling region. */ -extern sbitmap sched_insns; - extern void attach_life_info (void); -extern void sched_bbs_init (sbitmap); -extern void sched_bbs_finish (void); extern void remove_notes (rtx, rtx); extern rtx restore_other_notes (rtx, basic_block); @@ -296,11 +290,6 @@ extern struct ready_list ready; extern int max_issue (struct ready_list *, int, state_t, int *); -extern void sched_local_init (int); -extern void sched_local_finish (void); -extern void sched_data_update (sbitmap, basic_block, rtx); -extern void sched_data_finish (void); - extern void ebb_compute_jump_reg_dependencies (rtx, regset, regset, regset); extern edge find_fallthru_edge (basic_block); @@ -497,17 +486,22 @@ struct spec_info_def }; typedef struct spec_info_def *spec_info_t; +extern spec_info_t spec_info; + extern struct haifa_sched_info *current_sched_info; /* Indexed by INSN_UID, the collection of all data associated with a single instruction. */ -struct haifa_insn_data +struct _haifa_insn_data { /* A list of scheduled producers of the instruction. Links are being moved from LOG_LINKS to RESOLVED_DEPS during scheduling. */ rtx resolved_deps; + /* A priority for each insn. */ + int priority; + /* The minimum clock tick at which the insn becomes ready. This is used to note timing constraints for the insns in the pending list. */ int tick; @@ -521,15 +515,24 @@ struct haifa_insn_data short cost; + /* This weight is an estimation of the insn's contribution to + register pressure. */ + short reg_weight; + /* Set if there's DEF-USE dependence between some speculatively moved load insn and this one. */ unsigned int fed_by_spec_load : 1; unsigned int is_load_insn : 1; + /* Nonzero if priority has been computed already. */ + unsigned int priority_known : 1; + /* What speculations are necessary to apply to schedule the instruction. */ ds_t todo_spec; + /* What speculations were already applied. */ ds_t done_spec; + /* What speculations are checked by this instruction. */ ds_t check_spec; @@ -540,16 +543,28 @@ struct haifa_insn_data rtx orig_pat; }; -extern struct haifa_insn_data *h_i_d; +typedef struct _haifa_insn_data haifa_insn_data_def; +typedef haifa_insn_data_def *haifa_insn_data_t; + +DEF_VEC_O (haifa_insn_data_def); +DEF_VEC_ALLOC_O (haifa_insn_data_def, heap); + +extern VEC (haifa_insn_data_def, heap) *h_i_d; + +#define HID(INSN) (VEC_index (haifa_insn_data_def, h_i_d, INSN_UID (INSN))) /* Accessor macros for h_i_d. There are more in haifa-sched.c and sched-rgn.c. */ -#define RESOLVED_DEPS(INSN) (h_i_d[INSN_UID (INSN)].resolved_deps) -#define TODO_SPEC(INSN) (h_i_d[INSN_UID (INSN)].todo_spec) -#define DONE_SPEC(INSN) (h_i_d[INSN_UID (INSN)].done_spec) -#define CHECK_SPEC(INSN) (h_i_d[INSN_UID (INSN)].check_spec) -#define RECOVERY_BLOCK(INSN) (h_i_d[INSN_UID (INSN)].recovery_block) -#define ORIG_PAT(INSN) (h_i_d[INSN_UID (INSN)].orig_pat) +#define RESOLVED_DEPS(INSN) (HID (INSN)->resolved_deps) +#define INSN_PRIORITY(INSN) (HID (INSN)->priority) +#define INSN_REG_WEIGHT(INSN) (HID (INSN)->reg_weight) +#define INSN_PRIORITY_KNOWN(INSN) (HID (INSN)->priority_known) +#define INSN_COST(INSN) (HID (INSN)->cost) +#define TODO_SPEC(INSN) (HID (INSN)->todo_spec) +#define DONE_SPEC(INSN) (HID (INSN)->done_spec) +#define CHECK_SPEC(INSN) (HID (INSN)->check_spec) +#define RECOVERY_BLOCK(INSN) (HID (INSN)->recovery_block) +#define ORIG_PAT(INSN) (HID (INSN)->orig_pat) /* INSN is either a simple or a branchy speculation check. */ #define IS_SPECULATION_CHECK_P(INSN) (RECOVERY_BLOCK (INSN) != NULL) @@ -566,19 +581,6 @@ extern struct haifa_insn_data *h_i_d; #define IS_SPECULATION_BRANCHY_CHECK_P(INSN) \ (RECOVERY_BLOCK (INSN) != NULL && RECOVERY_BLOCK (INSN) != EXIT_BLOCK_PTR) -#define INSN_PRIORITY(INSN) (d_i_d[INSN_LUID (INSN)].priority) -#define INSN_PRIORITY_KNOWN(INSN) (d_i_d[INSN_LUID (INSN)].priority_known) - -/* !!! FIXME: Move this logic to haifa-sched.c: insn_cost (). */ -#define INSN_COST(INSN) __extension__ \ -(*({ short * _cost; \ - rtx const _insn = (INSN); \ - if (SEL_SCHED_P) \ - _cost = &(d_i_d[INSN_LUID (_insn)].cost); \ - else \ - _cost = &(h_i_d[INSN_UID (_insn)].cost); \ - _cost; })) - /* DEP_STATUS of the link encapsulates information, that is needed for speculative scheduling. Namely, it is 4 integers in the range [0, MAX_DEP_WEAK] and 3 bits. @@ -722,7 +724,8 @@ enum SCHED_FLAGS { enum SPEC_SCHED_FLAGS { COUNT_SPEC_IN_CRITICAL_PATH = 1, PREFER_NON_DATA_SPEC = COUNT_SPEC_IN_CRITICAL_PATH << 1, - PREFER_NON_CONTROL_SPEC = PREFER_NON_DATA_SPEC << 1 + PREFER_NON_CONTROL_SPEC = PREFER_NON_DATA_SPEC << 1, + SEL_SCHED_SPEC_DONT_CHECK_CONTROL = PREFER_NON_CONTROL_SPEC << 1 }; #ifndef __GNUC__ @@ -751,6 +754,10 @@ extern rtx ready_element (struct ready_list *, int); extern rtx *ready_lastpos (struct ready_list *); extern int try_ready (rtx); +extern void sched_extend_ready_list (int); +extern void sched_finish_ready_list (void); +extern void sched_change_pattern (rtx, rtx); +extern int sched_speculate_insn (rtx, ds_t, rtx *); extern void unlink_bb_notes (basic_block, basic_block); extern void add_block (basic_block, basic_block); extern rtx bb_note (basic_block); @@ -763,13 +770,13 @@ extern void check_reg_live (bool); extern void haifa_sched_init (void); extern void haifa_sched_finish (void); -extern void haifa_local_init (int); +extern dw_t dep_weak (ds_t); /* Functions in sched-rgn.c. */ extern void compute_priorities (void); extern void debug_dependencies (void); extern int contributes_to_priority (rtx, rtx); -extern void free_rgn_deps (bool); +extern void free_rgn_deps (void); extern void extend_rgns (int *, int *, sbitmap, int *); diff --git a/gcc/sched-rgn.c b/gcc/sched-rgn.c index 488d81f6708..f55edc35052 100644 --- a/gcc/sched-rgn.c +++ b/gcc/sched-rgn.c @@ -85,8 +85,8 @@ Software Foundation, 51 Franklin Street, Fifth Floor, Boston, MA #ifdef INSN_SCHEDULING /* Some accessor macros for h_i_d members only used within this file. */ -#define FED_BY_SPEC_LOAD(insn) (h_i_d[INSN_UID (insn)].fed_by_spec_load) -#define IS_LOAD_INSN(insn) (h_i_d[INSN_UID (insn)].is_load_insn) +#define FED_BY_SPEC_LOAD(INSN) (HID (INSN)->fed_by_spec_load) +#define IS_LOAD_INSN(INSN) (HID (insn)->is_load_insn) /* nr_inter/spec counts interblock/speculative motion for the function. */ static int nr_inter, nr_spec; @@ -130,8 +130,6 @@ static bool too_large (int, int *, int *); int current_nr_blocks; int current_blocks; -static int rgn_n_insns; - candidate *candidate_table; /* A speculative motion requires checking live information on the path @@ -451,6 +449,9 @@ find_single_block_region (bool ebbs_p) RGN_BLOCKS (nr_regions) = i; RGN_DONT_CALC_DEPS (nr_regions) = 0; RGN_HAS_REAL_EBB (nr_regions) = 0; + RGN_HAS_RENAMING_P (nr_regions) = 0; + RGN_WAS_PIPELINED_P (nr_regions) = 0; + for (bb = ebb_start; ; bb = bb->next_bb) { edge e; @@ -474,6 +475,12 @@ find_single_block_region (bool ebbs_p) if (e->probability <= probability_cutoff) break; } + + if (RGN_NR_BLOCKS (nr_regions) == 1) + RGN_NEEDS_GLOBAL_LIVE_UPDATE (nr_regions) = 0; + else + RGN_NEEDS_GLOBAL_LIVE_UPDATE (nr_regions) = 1; + ebb_start = bb; nr_regions++; } @@ -486,6 +493,10 @@ find_single_block_region (bool ebbs_p) RGN_BLOCKS (nr_regions) = nr_regions; RGN_DONT_CALC_DEPS (nr_regions) = 0; RGN_HAS_REAL_EBB (nr_regions) = 0; + RGN_HAS_RENAMING_P (nr_regions) = 0; + RGN_WAS_PIPELINED_P (nr_regions) = 0; + RGN_NEEDS_GLOBAL_LIVE_UPDATE (nr_regions) = 0; + CONTAINING_RGN (bb->index) = nr_regions; BLOCK_TO_BB (bb->index) = 0; nr_regions++; @@ -494,7 +505,7 @@ find_single_block_region (bool ebbs_p) /* Estimate number of the insns in the BB. */ static int -sched_rgn_estimate_number_of_insns (basic_block bb) +rgn_estimate_number_of_insns (basic_block bb) { return INSN_LUID (BB_END (bb)) - INSN_LUID (BB_HEAD (bb)); } @@ -937,6 +948,12 @@ haifa_find_rgns (void) RGN_BLOCKS (nr_regions) = idx++; RGN_DONT_CALC_DEPS (nr_regions) = 0; RGN_HAS_REAL_EBB (nr_regions) = 0; + RGN_HAS_RENAMING_P (nr_regions) = 0; + RGN_WAS_PIPELINED_P (nr_regions) = 0; + if (num_bbs == 1) + RGN_NEEDS_GLOBAL_LIVE_UPDATE (nr_regions) = 0; + else + RGN_NEEDS_GLOBAL_LIVE_UPDATE (nr_regions) = 1; CONTAINING_RGN (bb->index) = nr_regions; BLOCK_TO_BB (bb->index) = count = 0; @@ -1008,6 +1025,9 @@ haifa_find_rgns (void) RGN_BLOCKS (nr_regions) = idx++; RGN_DONT_CALC_DEPS (nr_regions) = 0; RGN_HAS_REAL_EBB (nr_regions) = 0; + RGN_HAS_RENAMING_P (nr_regions) = 0; + RGN_WAS_PIPELINED_P (nr_regions) = 0; + RGN_NEEDS_GLOBAL_LIVE_UPDATE (nr_regions) = 0; CONTAINING_RGN (bb->index) = nr_regions++; BLOCK_TO_BB (bb->index) = 0; } @@ -1137,7 +1157,8 @@ extend_rgns (int *degree, int *idxp, sbitmap header, int *loop_hdr) CFG should be traversed until no further changes are made. On each iteration the set of the region heads is extended (the set of those blocks that have max_hdr[bbi] == bbi). This set is upper bounded by the - set of all basic blocks, thus the algorithm is guaranteed to terminate. */ + set of all basic blocks, thus the algorithm is guaranteed to + terminate. */ while (rescan && iter < max_iter) { @@ -1254,6 +1275,8 @@ extend_rgns (int *degree, int *idxp, sbitmap header, int *loop_hdr) RGN_BLOCKS (nr_regions) = idx++; RGN_DONT_CALC_DEPS (nr_regions) = 0; RGN_HAS_REAL_EBB (nr_regions) = 0; + RGN_HAS_RENAMING_P (nr_regions) = 0; + RGN_WAS_PIPELINED_P (nr_regions) = 0; CONTAINING_RGN (bbn) = nr_regions; BLOCK_TO_BB (bbn) = 0; @@ -1280,6 +1303,7 @@ extend_rgns (int *degree, int *idxp, sbitmap header, int *loop_hdr) processed in the below cycle. */ { RGN_NR_BLOCKS (nr_regions) = 1; + RGN_NEEDS_GLOBAL_LIVE_UPDATE (nr_regions) = 0; nr_regions++; } @@ -1309,6 +1333,9 @@ extend_rgns (int *degree, int *idxp, sbitmap header, int *loop_hdr) RGN_NR_BLOCKS (nr_regions) = 1; RGN_DONT_CALC_DEPS (nr_regions) = 0; RGN_HAS_REAL_EBB (nr_regions) = 0; + RGN_HAS_RENAMING_P (nr_regions) = 0; + RGN_WAS_PIPELINED_P (nr_regions) = 0; + RGN_NEEDS_GLOBAL_LIVE_UPDATE (nr_regions) = 0; nr_regions++; } @@ -1323,6 +1350,10 @@ extend_rgns (int *degree, int *idxp, sbitmap header, int *loop_hdr) if (!large) { RGN_NR_BLOCKS (nr_regions) = num_bbs; + if (num_bbs == 1) + RGN_NEEDS_GLOBAL_LIVE_UPDATE (nr_regions) = 0; + else + RGN_NEEDS_GLOBAL_LIVE_UPDATE (nr_regions) = 1; nr_regions++; } } @@ -2029,13 +2060,13 @@ static int rgn_rank (rtx, rtx); static void compute_jump_reg_dependencies (rtx, regset, regset, regset); /* Functions for speculative scheduling. */ -static void add_remove_insn (rtx, int); -static void add_block1 (basic_block, basic_block); -static void fix_recovery_cfg (int, int, int); +static void rgn_add_remove_insn (rtx, int); +static void rgn_add_block (basic_block, basic_block); +static void rgn_fix_recovery_cfg (int, int, int); static basic_block advance_target_bb (basic_block, rtx); static void check_dead_notes1 (int, sbitmap); #ifdef ENABLE_CHECKING -static int region_head_or_leaf_p (basic_block, int); +static int rgn_region_head_or_leaf_p (basic_block, int); #endif /* Return nonzero if there are more insns that should be scheduled. */ @@ -2264,24 +2295,9 @@ compute_jump_reg_dependencies (rtx insn ATTRIBUTE_UNUSED, add_branch_dependences. */ } -struct common_sched_info_def rgn_common_sched_info = - { - fix_recovery_cfg, add_block1, - sched_rgn_estimate_number_of_insns, - NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, -#ifdef ENABLE_CHECKING - region_head_or_leaf_p, -#endif - 1, -#ifdef ENABLE_CHECKING - 1, -#else - 0, -#endif - SCHED_RGN_PASS - }; +static struct common_sched_info_def rgn_common_sched_info; -struct sched_deps_info_def rgn_sched_deps_info = +static const struct sched_deps_info_def rgn_const_sched_deps_info = { compute_jump_reg_dependencies, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, @@ -2289,7 +2305,7 @@ struct sched_deps_info_def rgn_sched_deps_info = 0, 0, 0 }; -struct sched_deps_info_def rgn_after_reload_sched_deps_info = +static const struct sched_deps_info_def rgn_const_sel_sched_deps_info = { compute_jump_reg_dependencies, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, @@ -2297,10 +2313,12 @@ struct sched_deps_info_def rgn_after_reload_sched_deps_info = 0, 0, 0 }; +static struct sched_deps_info_def rgn_sched_deps_info; + /* Used in schedule_insns to initialize current_sched_info for scheduling regions (or single basic blocks). */ -static struct haifa_sched_info region_sched_info = +static const struct haifa_sched_info rgn_const_sched_info = { init_ready_list, can_schedule_ready_p, @@ -2314,11 +2332,13 @@ static struct haifa_sched_info region_sched_info = NULL, NULL, 0, 0, - add_remove_insn, + rgn_add_remove_insn, begin_schedule_ready, advance_target_bb }; +static struct haifa_sched_info rgn_sched_info; + /* Determine if PAT sets a CLASS_LIKELY_SPILLED_P register. */ static bool @@ -2754,7 +2774,7 @@ sched_is_disabled_for_current_region_p (void) The Haifa scheduler does this on the fly when scheduling, so this function is supposed to be called from sel-sched only. */ void -free_rgn_deps (bool free_insn_depend) +free_rgn_deps (void) { int bb; @@ -2779,21 +2799,14 @@ free_rgn_deps (bool free_insn_depend) } else { - if (free_insn_depend) - { - gcc_assert (!LOG_LINKS (insn)); - if (INSN_DEPEND (insn)) - free_INSN_LIST_list (&INSN_DEPEND (insn)); - } - else - { - free_INSN_LIST_list (&LOG_LINKS (insn)); - } + free_INSN_LIST_list (&LOG_LINKS (insn)); + free_INSN_LIST_list (&INSN_DEPEND (insn)); } } } } +static int rgn_n_insns; /* Compute insn priority for a current region. */ void @@ -2826,13 +2839,17 @@ schedule_region (int rgn) rgn_n_insns = 0; - if (sched_rgn_local_init (rgn, true)) + rgn_setup_region (rgn); + + if (sched_rgn_local_preinit (rgn)) return; + sched_rgn_local_init (rgn); + /* Set priorities. */ compute_priorities (); - haifa_local_init (rgn_n_insns); + sched_extend_ready_list (rgn_n_insns); /* Now we can schedule all blocks. */ for (bb = 0; bb < current_nr_blocks; bb++) @@ -2876,6 +2893,8 @@ schedule_region (int rgn) /* Sanity check: verify that all region insns were scheduled. */ gcc_assert (sched_rgn_n_insns == rgn_n_insns); + sched_finish_ready_list (); + /* Done with this region. */ sched_rgn_local_finish (); } @@ -2929,9 +2948,11 @@ sched_rgn_init (bool single_blocks_p, bool ebbs_p) if (!SEL_SCHED_P || !flag_sel_sched_pipelining_outer_loops) free_dominance_info (CDI_DOMINATORS); } - RGN_BLOCKS (nr_regions) = RGN_BLOCKS (nr_regions - 1) + - RGN_NR_BLOCKS (nr_regions - 1); + gcc_assert (0 < nr_regions && nr_regions <= n_basic_blocks); + + RGN_BLOCKS (nr_regions) = (RGN_BLOCKS (nr_regions - 1) + + RGN_NR_BLOCKS (nr_regions - 1)); if (CHECK_DEAD_NOTES) { @@ -2955,6 +2976,7 @@ sched_rgn_finish (void) int any_large_regions; basic_block bb; int rgn; + bool check_live_info; /* Update life analysis for the subroutine. Do single block regions first so that we can verify that live_at_start didn't change. Then @@ -2984,12 +3006,24 @@ sched_rgn_finish (void) blocks = sbitmap_alloc (last_basic_block); sbitmap_zero (blocks); + /* We may have problems with conditional sets. Avoid them. */ + check_live_info = (!reload_completed + || (reload_completed && CHECK_DEAD_NOTES)); + + if (SEL_SCHED_P) + /* During selective scheduling CFG is being modified in various + ways and it is really difficult to keep track of liveness info + for now. */ + check_live_info = false; + /* Update life information. For regions consisting of multiple blocks we've possibly done interblock scheduling that affects global liveness. For regions consisting of single blocks we need to do only local liveness. */ for (rgn = 0; rgn < nr_regions; rgn++) - if (RGN_NR_BLOCKS (rgn) > 1 + if (!check_live_info + || RGN_NR_BLOCKS (rgn) > 1 + || RGN_NEEDS_GLOBAL_LIVE_UPDATE (rgn) /* Or the only block of this region has been split. */ || RGN_HAS_REAL_EBB (rgn) /* New blocks (e.g. recovery blocks) should be processed @@ -3007,6 +3041,7 @@ sched_rgn_finish (void) update_life_info (blocks, UPDATE_LIFE_LOCAL, (reload_completed ? PROP_DEATH_NOTES : (PROP_DEATH_NOTES | PROP_REG_INFO))); + if (any_large_regions) { update_life_info (large_region_blocks, UPDATE_LIFE_GLOBAL, @@ -3014,27 +3049,26 @@ sched_rgn_finish (void) : (PROP_DEATH_NOTES | PROP_REG_INFO))); #ifdef ENABLE_CHECKING - if (/* We may have problems with conditional sets. Avoid them. */ - !reload_completed - || (reload_completed && CHECK_DEAD_NOTES)) + if (check_live_info) check_reg_live (true); #endif } sbitmap_free (large_region_blocks); - if (CHECK_DEAD_NOTES) + if (CHECK_DEAD_NOTES && !SEL_SCHED_P) { /* Verify the counts of basic block notes in single basic block regions. */ for (rgn = 0; rgn < nr_regions; rgn++) - if (RGN_NR_BLOCKS (rgn) == 1) + if (RGN_NR_BLOCKS (rgn) == 1 && !RGN_NEEDS_GLOBAL_LIVE_UPDATE (rgn)) { sbitmap_zero (blocks); SET_BIT (blocks, rgn_bb_table[RGN_BLOCKS (rgn)]); -/* gcc_assert (deaths_in_region[rgn] - == count_or_remove_death_notes (blocks, 0)); */ + gcc_assert (RGN_HAS_RENAMING_P (rgn) + || (deaths_in_region[rgn] + == count_or_remove_death_notes (blocks, 0))); } free (deaths_in_region); } @@ -3048,9 +3082,7 @@ sched_rgn_finish (void) if (sched_verbose) { - if ((/* FORCE_REGIONS */ - true - || reload_completed == 0) + if (reload_completed == 0 && flag_schedule_interblock) { fprintf (sched_dump, @@ -3080,10 +3112,10 @@ sched_rgn_finish (void) ebb_head = NULL; } -/* Ugly hack: do preinit of region structures, equally suitable for - selective and haifa schedulers. */ -bool -sched_rgn_local_preinit (int rgn) +/* Setup global variables like CURRENT_BLOCKS and CURRENT_NR_BLOCK to + point to the region RGN. */ +void +rgn_setup_region (int rgn) { int bb; @@ -3097,37 +3129,24 @@ sched_rgn_local_preinit (int rgn) ebb_head = xrealloc (ebb_head, (current_nr_blocks + 1) * sizeof (*ebb_head)); for (bb = 0; bb <= current_nr_blocks; bb++) ebb_head[bb] = current_blocks + bb; +} +/* Ugly hack: do preinit of region structures, equally suitable for + selective and haifa schedulers. */ +bool +sched_rgn_local_preinit (int rgn) +{ /* Don't schedule region that is marked by NOTE_DISABLE_SCHED_OF_BLOCK. */ if (sched_is_disabled_for_current_region_p ()) return true; - return false; -} - -/* Init region data structures. Returns true if this region should - not be scheduled. */ -bool -sched_rgn_local_init (int rgn, bool do_preinit_p) -{ - int bb; - if (do_preinit_p && sched_rgn_local_preinit (rgn)) - return true; - - if (!RGN_DONT_CALC_DEPS (rgn) /*&& !SEL_SCHED_P*/) + if (!RGN_DONT_CALC_DEPS (rgn)) { - struct sched_deps_info_def *old_deps_info = sched_deps_info; - + int bb; + if (SEL_SCHED_P) - { - sched_deps_info = &rgn_after_reload_sched_deps_info; - sched_emulate_haifa_p = 1; -#if 0 - if (sched_deps_info->use_cselib) - cselib_init (true); -#endif - } + sched_emulate_haifa_p = 1; init_deps_global (); @@ -3136,7 +3155,8 @@ sched_rgn_local_init (int rgn, bool do_preinit_p) for (bb = 0; bb < current_nr_blocks; bb++) init_deps (bb_deps + bb); - ref_counts = xcalloc (sched_max_uid, sizeof (*ref_counts)); + /* Initialize array used in add_branch_dependencies (). */ + ref_counts = xcalloc (get_max_uid () + 1, sizeof (*ref_counts)); /* Compute LOG_LINKS. */ for (bb = 0; bb < current_nr_blocks; bb++) @@ -3152,7 +3172,8 @@ sched_rgn_local_init (int rgn, bool do_preinit_p) rtx head, tail; gcc_assert (EBB_FIRST_BB (bb) == EBB_LAST_BB (bb)); - get_ebb_head_tail (EBB_FIRST_BB (bb), EBB_LAST_BB (bb), &head, &tail); + get_ebb_head_tail (EBB_FIRST_BB (bb), EBB_LAST_BB (bb), &head, + &tail); compute_forward_dependences (head, tail); @@ -3162,11 +3183,6 @@ sched_rgn_local_init (int rgn, bool do_preinit_p) free_pending_lists (); -#if 0 - if (sched_verbose >= 5 && sched_emulate_haifa_p) - debug_dependencies (); -#endif - finish_deps_global (); free (bb_deps); @@ -3175,19 +3191,22 @@ sched_rgn_local_init (int rgn, bool do_preinit_p) RGN_DONT_CALC_DEPS (rgn) = 1; if (SEL_SCHED_P) - { -#if 0 - if (sched_deps_info->use_cselib) - cselib_finish (); -#endif - sched_emulate_haifa_p = 0; - sched_deps_info = old_deps_info; - } + sched_emulate_haifa_p = 0; } else /* (This is a recovery block. It is always a single block region.) OR (We use selective scheduling.) */ gcc_assert (current_nr_blocks == 1 || SEL_SCHED_P); + + return false; +} + +/* Init region data structures. Returns true if this region should + not be scheduled. */ +void +sched_rgn_local_init (int rgn) +{ + int bb; /* Compute interblock info: probabilities, split-edges, dominators, etc. */ if (current_nr_blocks > 1 && !SEL_SCHED_P) @@ -3242,8 +3261,6 @@ sched_rgn_local_init (int rgn, bool do_preinit_p) e->aux = NULL; } } - - return false; } /* Free data computed for the finished region. */ @@ -3257,20 +3274,57 @@ sched_rgn_local_free (void) free (rgn_edges); } +/* Free data computed for the finished region. */ void sched_rgn_local_finish (void) { - sched_local_finish (); - if (current_nr_blocks > 1 && !SEL_SCHED_P) { sched_rgn_local_free (); } } -/* The one entry point in this file. */ +/* Setup scheduler infos. */ +void +rgn_setup_common_sched_info (void) +{ + memcpy (&rgn_common_sched_info, &haifa_common_sched_info, + sizeof (rgn_common_sched_info)); + + rgn_common_sched_info.fix_recovery_cfg = rgn_fix_recovery_cfg; + rgn_common_sched_info.add_block = rgn_add_block; + rgn_common_sched_info.estimate_number_of_insns + = rgn_estimate_number_of_insns; + rgn_common_sched_info.region_head_or_leaf_p = rgn_region_head_or_leaf_p; + rgn_common_sched_info.use_glat = 1; +#ifdef ENABLE_CHECKING + rgn_common_sched_info.detach_life_info = 1; +#else + rgn_common_sched_info.detach_life_info = 0; +#endif + rgn_common_sched_info.sched_pass_id = SCHED_RGN_PASS; + + common_sched_info = &rgn_common_sched_info; +} void +rgn_setup_sched_infos (void) +{ + if (!SEL_SCHED_P) + memcpy (&rgn_sched_deps_info, &rgn_const_sched_deps_info, + sizeof (rgn_sched_deps_info)); + else + memcpy (&rgn_sched_deps_info, &rgn_const_sel_sched_deps_info, + sizeof (rgn_sched_deps_info)); + + sched_deps_info = &rgn_sched_deps_info; + + memcpy (&rgn_sched_info, &rgn_const_sched_info, sizeof (rgn_sched_info)); + current_sched_info = &rgn_sched_info; +} + +/* The one entry point in this file. */ +void schedule_insns (void) { int rgn; @@ -3280,14 +3334,12 @@ schedule_insns (void) if (n_basic_blocks == NUM_FIXED_BLOCKS) return; - common_sched_info = &rgn_common_sched_info; - sched_deps_info = &rgn_sched_deps_info; - current_sched_info = ®ion_sched_info; + rgn_setup_common_sched_info (); + rgn_setup_sched_infos (); haifa_sched_init (); - /* FORCE_REGIONS */ - sched_rgn_init (false /*reload_completed*/, false); + sched_rgn_init (reload_completed, false); /* Schedule every region in the subroutine. */ for (rgn = 0; rgn < nr_regions; rgn++) @@ -3301,7 +3353,7 @@ schedule_insns (void) /* INSN has been added to/removed from current region. */ static void -add_remove_insn (rtx insn, int remove_p) +rgn_add_remove_insn (rtx insn, int remove_p) { if (!remove_p) rgn_n_insns++; @@ -3329,7 +3381,7 @@ extend_regions (void) /* BB was added to ebb after AFTER. */ static void -add_block1 (basic_block bb, basic_block after) +rgn_add_block (basic_block bb, basic_block after) { extend_regions (); @@ -3344,6 +3396,9 @@ add_block1 (basic_block bb, basic_block after) RGN_NR_BLOCKS (nr_regions) = 1; RGN_DONT_CALC_DEPS (nr_regions) = after == EXIT_BLOCK_PTR; RGN_HAS_REAL_EBB (nr_regions) = 0; + RGN_HAS_RENAMING_P (nr_regions) = 0; + RGN_WAS_PIPELINED_P (nr_regions) = 0; + RGN_NEEDS_GLOBAL_LIVE_UPDATE (nr_regions) = 0; CONTAINING_RGN (bb->index) = nr_regions; BLOCK_TO_BB (bb->index) = 0; @@ -3413,9 +3468,6 @@ add_block1 (basic_block bb, basic_block after) for (++i; i <= nr_regions; i++) RGN_BLOCKS (i)++; - - /* We don't need to call check_dead_notes1 () because this new block - is just a split of the old. We don't want to count anything twice. */ } } @@ -3423,7 +3475,7 @@ add_block1 (basic_block bb, basic_block after) For parameter meaning please refer to sched-int.h: struct sched_info: fix_recovery_cfg. */ static void -fix_recovery_cfg (int bbi, int check_bbi, int check_bb_nexti) +rgn_fix_recovery_cfg (int bbi, int check_bbi, int check_bb_nexti) { int old_pos, new_pos, i; @@ -3484,7 +3536,7 @@ check_dead_notes1 (int rgn, sbitmap blocks) current region. For more information please refer to sched-int.h: struct sched_info: region_head_or_leaf_p. */ static int -region_head_or_leaf_p (basic_block bb, int leaf_p) +rgn_region_head_or_leaf_p (basic_block bb, int leaf_p) { if (!leaf_p) return bb->index == rgn_bb_table[RGN_BLOCKS (CONTAINING_RGN (bb->index))]; @@ -3496,6 +3548,11 @@ region_head_or_leaf_p (basic_block bb, int leaf_p) i = CONTAINING_RGN (bb->index); + if (SEL_SCHED_P) + /* Pipelining can make a register to become live at the end of the + cycle. */ + return 0; + FOR_EACH_EDGE (e, ei, bb->succs) if (e->dest != EXIT_BLOCK_PTR && CONTAINING_RGN (e->dest->index) == i diff --git a/gcc/sched-rgn.h b/gcc/sched-rgn.h index 8a9d2aab8e1..0a6a27fcacf 100644 --- a/gcc/sched-rgn.h +++ b/gcc/sched-rgn.h @@ -22,9 +22,6 @@ Software Foundation, 51 Franklin Street, Fifth Floor, Boston, MA #ifndef GCC_SCHED_RGN_H #define GCC_SCHED_RGN_H -extern struct common_sched_info_def rgn_common_sched_info; -extern struct sched_deps_info_def rgn_sched_deps_info; - /* A region is the main entity for interblock scheduling: insns are allowed to move between blocks in the same region, along control flow graph edges, in the 'up' direction. */ @@ -39,6 +36,9 @@ typedef struct unsigned int dont_calc_deps : 1; /* This region has at least one non-trivial ebb. */ unsigned int has_real_ebb : 1; + unsigned int has_renaming_p : 1; + unsigned int was_pipelined_p : 1; + unsigned int needs_global_live_update : 1; } region; @@ -52,6 +52,10 @@ extern int *containing_rgn; #define RGN_BLOCKS(rgn) (rgn_table[rgn].rgn_blocks) #define RGN_DONT_CALC_DEPS(rgn) (rgn_table[rgn].dont_calc_deps) #define RGN_HAS_REAL_EBB(rgn) (rgn_table[rgn].has_real_ebb) +#define RGN_HAS_RENAMING_P(RGN) (rgn_table[RGN].has_renaming_p) +#define RGN_WAS_PIPELINED_P(RGN) (rgn_table[RGN].was_pipelined_p) +#define RGN_NEEDS_GLOBAL_LIVE_UPDATE(RGN) \ + (rgn_table[RGN].needs_global_live_update) #define BLOCK_TO_BB(block) (block_to_bb[block]) #define CONTAINING_RGN(block) (containing_rgn[block]) @@ -67,8 +71,9 @@ extern int current_blocks; extern void sched_rgn_init (bool, bool); extern void sched_rgn_finish (void); +extern void rgn_setup_region (int); extern bool sched_rgn_local_preinit (int); -extern bool sched_rgn_local_init (int, bool); +extern void sched_rgn_local_init (int); extern void sched_rgn_local_finish (void); extern void sched_rgn_local_free (void); extern void extend_regions (void); @@ -78,6 +83,9 @@ extern void free_trg_info (void); extern int check_live (rtx, int); extern void update_live (rtx, int); +extern void rgn_setup_common_sched_info (void); +extern void rgn_setup_sched_infos (void); + extern void debug_regions (void); extern void debug_region (int); extern void dump_region_dot (FILE *, int); diff --git a/gcc/sched-vis.c b/gcc/sched-vis.c index 427a3daaf6b..e82023fca24 100644 --- a/gcc/sched-vis.c +++ b/gcc/sched-vis.c @@ -694,7 +694,7 @@ dump_insn_slim_1 (FILE *f, rtx x) { char t[BUF_LEN + 32]; - print_insn (t, x, 1); + print_insn (t, x, 0); fputs (t, f); } diff --git a/gcc/sel-sched-dump.c b/gcc/sel-sched-dump.c index 1731429ec62..790674d4bfc 100644 --- a/gcc/sel-sched-dump.c +++ b/gcc/sel-sched-dump.c @@ -70,7 +70,7 @@ static bool sel_dump_cfg_p; static bool sel_pipelining_verbose_p; /* Variables that are used to build the cfg dump file name. */ -static const char * const sel_debug_cfg_root = "./"; +static const char * const sel_debug_cfg_root = "./dot"; static const char * const sel_debug_cfg_root_postfix_default = ""; static const char *sel_debug_cfg_root_postfix = ""; static int sel_dump_cfg_fileno = -1; @@ -81,12 +81,6 @@ static int sel_debug_cfg_fileno = -1; files. */ bool sched_dump_to_dot_p = false; -/* Controls how an insn should be dumped. */ -static int dump_insn_flags = (DUMP_INSN_UID | DUMP_INSN_BBN | DUMP_INSN_COUNT); - -/* Controls how an insn from RHS should be dumped. */ -static int dump_rhs_insn_flags = DUMP_INSN_UID | DUMP_INSN_COUNT; - /* Controls how insns from a fence list should be dumped. */ static int dump_flist_insn_flags = (DUMP_INSN_UID | DUMP_INSN_BBN | DUMP_INSN_SEQNO); @@ -225,72 +219,277 @@ sel_print_rtl (rtx x) } } -/* Dump insn I honoring FLAGS. */ +static int dump_all = 0; + void -dump_insn_1 (insn_t i, int flags) +dump_insn_rtx_1 (rtx insn, int flags) { - if (print_block) + int all; + + if (!print_block) + return; + + all = (flags & 1) | dump_all; + if (all) + flags |= DUMP_INSN_RTX_ALL; + + line_start (); + print ("("); + + if (flags & DUMP_INSN_RTX_UID) + print ("%d;", INSN_UID (insn)); + + if (flags & DUMP_INSN_RTX_PATTERN) { - line_start (); + char buf[2048]; - if (!sched_dump_to_dot_p) - print ("("); + print_insn (buf, insn, 0); + print ("%s;", buf); + } - if (flags & DUMP_INSN_PATTERN) - { - char buf[2048]; + if (flags & DUMP_INSN_RTX_BBN) + { + basic_block bb = BLOCK_FOR_INSN (insn); - print_insn (buf, i, 0); - print ("pat:%s;", buf); - } - else if (flags & DUMP_INSN_UID) - print ("uid:%d;", INSN_UID (i)); + print ("bb:%d;", bb != NULL ? bb->index : -1); + } - if (flags & DUMP_INSN_BBN) - { - basic_block bb = BLOCK_FOR_INSN (i); + print (")"); + line_finish (); +} - print ("bb:%d;", bb ? bb->index : -1); - } +static int dump_insn_rtx_flags = DUMP_INSN_RTX_PATTERN; - if (flags & DUMP_INSN_SEQNO) - print ("seqno:%d;", INSN_SEQNO (i)); +void +dump_insn_rtx (rtx insn) +{ + dump_insn_rtx_1 (insn, dump_insn_rtx_flags); +} - if (flags & DUMP_INSN_COUNT) - print ("count:%d;", VINSN_COUNT (INSN_VI (i))); +static int debug_insn_rtx_flags = 1; - if (flags & DUMP_INSN_CYCLE) - print ("cycle:%d;", INSN_SCHED_CYCLE (i)); +void +debug_insn_rtx (rtx insn) +{ + switch_dump (); + dump_insn_rtx_1 (insn, debug_insn_rtx_flags); + switch_dump (); +} - if (!sched_dump_to_dot_p) - print (")"); +void +dump_vinsn_1 (vinsn_t vi, int flags) +{ + int all; - line_finish (); + if (!print_block) + return; + + all = (flags & 1) | dump_all; + if (all) + flags |= DUMP_VINSN_ALL; + + line_start (); + print ("("); + + if (flags & DUMP_VINSN_INSN_RTX) + dump_insn_rtx_1 (VINSN_INSN_RTX (vi), dump_insn_rtx_flags | all); + + if (flags & DUMP_VINSN_TYPE) + print ("type:%s;", GET_RTX_NAME (VINSN_TYPE (vi))); + + if (flags & DUMP_VINSN_COUNT) + print ("count:%d;", VINSN_COUNT (vi)); + + if (flags & DUMP_VINSN_COST) + { + int cost = vi->cost; + + if (cost != -1) + print ("cost:%d;", cost); } + + print (")"); + line_finish (); } -/* Dump insn I with default flags. */ +static int dump_vinsn_flags = DUMP_VINSN_INSN_RTX | DUMP_VINSN_TYPE; + void -dump_insn (insn_t i) +dump_vinsn (vinsn_t vi) { - dump_insn_1 (i, dump_insn_flags); + dump_vinsn_1 (vi, dump_vinsn_flags); +} + +static int debug_vinsn_flags = 1; + +void +debug_vinsn (vinsn_t vi) +{ + switch_dump (); + dump_vinsn_1 (vi, debug_vinsn_flags); + switch_dump (); } /* Dump RHS. */ void +dump_expr_1 (expr_t expr, int flags) +{ + int all; + + if (!print_block) + return; + + all = (flags & 1) | dump_all; + if (all) + flags |= DUMP_EXPR_ALL; + + line_start (); + print ("["); + + if (flags & DUMP_EXPR_VINSN) + dump_vinsn_1 (EXPR_VINSN (expr), dump_vinsn_flags | all); + + if (flags & DUMP_EXPR_SPEC) + { + int spec = EXPR_SPEC (expr); + + if (spec != 0) + print ("spec:%d;", spec); + } + + if (flags & DUMP_EXPR_PRIORITY) + print ("prio:%d;", EXPR_PRIORITY (expr)); + + if (flags & DUMP_EXPR_SCHED_TIMES) + { + int times = EXPR_SCHED_TIMES (expr); + + if (times != 0) + print ("times:%d;", times); + } + + if (flags & DUMP_EXPR_SPEC_DONE_DS) + { + ds_t spec_done_ds = EXPR_SPEC_DONE_DS (expr); + + if (spec_done_ds != 0) + print ("ds:%d;", spec_done_ds); + } + + print ("]"); + line_finish (); +} + +static int dump_expr_flags = DUMP_EXPR_ALL; + +void +dump_expr (expr_t expr) +{ + dump_expr_1 (expr, dump_expr_flags); +} + +static int debug_expr_flags = 1; + +void +debug_expr (expr_t expr) +{ + switch_dump (); + dump_expr_1 (expr, debug_expr_flags); + switch_dump (); +} + +/* Obsolete. */ +void dump_rhs (rhs_t rhs) { - if (print_block) + dump_expr (rhs); +} + +/* Obsolete. */ +void +debug_rhs (rhs_t rhs) +{ + debug_expr (rhs); +} + +/* Dump insn I honoring FLAGS. */ +void +dump_insn_1 (insn_t i, int flags) +{ + int all; + + if (!print_block) + return; + + all = (flags & 1) | dump_all; + if (all) + flags |= DUMP_INSN_ALL; + + line_start (); + + if (!sched_dump_to_dot_p) + print ("("); + + if (flags & DUMP_INSN_ASM_P) + flags = flags; + + if (flags & DUMP_INSN_SCHED_NEXT) + flags = flags; + + if (flags & DUMP_INSN_EXPR) { - insn_t i = RHS_INSN (rhs); - - line_start (); - print ("["); - dump_insn_1 (i, dump_rhs_insn_flags | DUMP_INSN_PATTERN); - print (";as_rhs:%d", RHS_SCHEDULE_AS_RHS (rhs)); - print (";spec:%d]", RHS_SPEC (rhs)); - line_finish (); + dump_expr_1 (INSN_EXPR (i), dump_expr_flags | all); + print (";"); + } + else if (flags & DUMP_INSN_PATTERN) + { + dump_insn_rtx_1 (i, DUMP_INSN_RTX_PATTERN | all); + print (";"); } + else if (flags & DUMP_INSN_UID) + print ("uid:%d;", INSN_UID (i)); + + if (flags & DUMP_INSN_AV) + flags = flags; + + if (flags & DUMP_INSN_SEQNO) + print ("seqno:%d;", INSN_SEQNO (i)); + + if (flags & DUMP_INSN_AFTER_STALL_P) + flags = flags; + + if (flags & DUMP_INSN_SCHED_CYCLE) + { + int cycle = INSN_SCHED_CYCLE (i); + + if (cycle != 0) + print ("cycle:%d;", cycle); + } + + if (!sched_dump_to_dot_p) + print (")"); + + line_finish (); +} + +/* Controls how an insn should be dumped. It can be changed from debugger. */ +static int dump_insn_flags = (DUMP_INSN_EXPR | DUMP_INSN_SCHED_CYCLE); + +/* Dump insn I with default flags. */ +void +dump_insn (insn_t i) +{ + dump_insn_1 (i, dump_insn_flags); +} + +static int debug_insn_flags = 1; + +void +debug_insn (insn_t insn) +{ + switch_dump (); + dump_insn_1 (insn, debug_insn_flags); + switch_dump (); } /* Dump used regs from USED_REGS bitmap. */ @@ -474,6 +673,7 @@ dump_hard_reg_set (const char *prefix, HARD_REG_SET set) } } +#if 0 /* Pretty print INSN. This is used as a hook. */ const char * sel_print_insn (rtx insn, int aligned ATTRIBUTE_UNUSED) @@ -499,6 +699,7 @@ sel_print_insn (rtx insn, int aligned ATTRIBUTE_UNUSED) return buf; } +#endif /* Functions for pretty printing of CFG. */ @@ -565,8 +766,8 @@ sel_dump_cfg_insn (insn_t insn, int flags) int insn_flags = DUMP_INSN_UID | DUMP_INSN_PATTERN; if ((flags & SEL_DUMP_CFG_INSN_SEQNO) - && TEST_BIT (sched_insns, INSN_UID (insn))) - insn_flags |= DUMP_INSN_SEQNO | DUMP_INSN_CYCLE; + && INSN_LUID (insn) > 0) + insn_flags |= DUMP_INSN_SEQNO | DUMP_INSN_SCHED_CYCLE | DUMP_INSN_EXPR; dump_insn_1 (insn, insn_flags); } @@ -661,12 +862,42 @@ sel_dump_cfg_2 (FILE *f, int flags) color = ""; if ((flags & SEL_DUMP_CFG_FENCES) - && in_region_p && !bb_empty_p (bb) - && IN_CURRENT_FENCE_P (NEXT_INSN (bb_note (bb)))) + && in_region_p) { - /*style = "style = dotted, ";*/ style = ""; - color = "color = red, "; + + if (!sel_bb_empty_p (bb)) + { + bool first_p = true; + insn_t tail = BB_END (bb); + insn_t cur_insn; + + cur_insn = bb_note (bb); + + do + { + fence_t fence; + + cur_insn = NEXT_INSN (cur_insn); + fence = flist_lookup (fences, cur_insn); + + if (fence != NULL) + { + if (!FENCE_SCHEDULED_SOMETHING (fence)) + { + if (first_p) + color = "color = red, "; + else + color = "color = yellow, "; + } + else + color = "color = blue, "; + } + + first_p = false; + } + while (cur_insn != tail); + } } else if (!full_p) style = "style = dashed, "; @@ -700,7 +931,7 @@ sel_dump_cfg_2 (FILE *f, int flags) if (full_p && (flags & SEL_DUMP_CFG_AV_SET) && in_current_region_p (bb) - && !bb_empty_p (bb)) + && !sel_bb_empty_p (bb)) { insn_t head = NEXT_INSN (bb_note (bb)); @@ -716,7 +947,7 @@ sel_dump_cfg_2 (FILE *f, int flags) } if ((flags & SEL_DUMP_CFG_LV_SET) - && !bb_empty_p (bb)) + && !sel_bb_empty_p (bb)) { insn_t head; insn_t tail; @@ -731,6 +962,20 @@ sel_dump_cfg_2 (FILE *f, int flags) fprintf (f, "!!! Wrong LV_SET"); } + if (flags & SEL_DUMP_CFG_BB_LIVE) + { + regset rs; + + fprintf (f, "|"); + + rs = bb->il.rtl->global_live_at_start; + + if (rs != NULL) + dump_lv_set (rs); + else + fprintf (f, "!!! Wrong live_at_start"); + } + if (full_p && (flags & SEL_DUMP_CFG_BB_INSNS)) { @@ -744,6 +989,20 @@ sel_dump_cfg_2 (FILE *f, int flags) } } + if (flags & SEL_DUMP_CFG_BB_LIVE) + { + regset rs; + + fprintf (f, "|"); + + rs = bb->il.rtl->global_live_at_end; + + if (rs != NULL) + dump_lv_set (rs); + else + fprintf (f, "!!! Wrong live_at_end"); + } + fprintf (f, "}\"];\n"); FOR_EACH_EDGE (e, ei, bb->succs) @@ -868,14 +1127,6 @@ hard_regno_rename_ok (int i ATTRIBUTE_UNUSED, int j ATTRIBUTE_UNUSED) return HARD_REGNO_RENAME_OK (i, j); } -void -debug_rhs (rhs_t r) -{ - switch_dump (); - dump_rhs (r); - switch_dump (); -} - /* Dumps av_set AV to stderr. */ void debug_av_set (av_set_t av) diff --git a/gcc/sel-sched-dump.h b/gcc/sel-sched-dump.h index 910b005e156..4bf1d3e8c77 100644 --- a/gcc/sel-sched-dump.h +++ b/gcc/sel-sched-dump.h @@ -36,6 +36,7 @@ Software Foundation, 51 Franklin Street, Fifth Floor, Boston, MA #define SEL_DUMP_CFG_INSN_SEQNO (128) #define SEL_DUMP_CFG_INSN_FLAGS (0) #define SEL_DUMP_CFG_FUNCTION_NAME (256) +#define SEL_DUMP_CFG_BB_LIVE (512) /* The default flags for cfg dumping. */ #define SEL_DUMP_CFG_FLAGS (SEL_DUMP_CFG_CURRENT_REGION \ | SEL_DUMP_CFG_BB_NOTES_LIST \ @@ -46,17 +47,90 @@ Software Foundation, 51 Franklin Street, Fifth Floor, Boston, MA | SEL_DUMP_CFG_INSN_SEQNO \ | SEL_DUMP_CFG_INSN_FLAGS) +enum _dump_insn_rtx + { + DUMP_INSN_RTX_UID = 2, + DUMP_INSN_RTX_PATTERN = 4, + DUMP_INSN_RTX_BBN = 8, + + DUMP_INSN_RTX_ALL = (DUMP_INSN_RTX_UID | DUMP_INSN_RTX_PATTERN + | DUMP_INSN_RTX_BBN) + }; + +extern void dump_insn_rtx_1 (rtx, int); +extern void dump_insn_rtx (rtx); +extern void debug_insn_rtx (rtx); + +enum _dump_idata + { + DUMP_IDATA_TYPE = 2, + DUMP_IDATA_LHS = 4, + DUMP_IDATA_RHS = 8, + DUMP_IDATA_REG_SETS = 16, + DUMP_IDATA_REG_USES = 32, + + DUMP_IDATA_ALL = (DUMP_IDATA_TYPE | DUMP_IDATA_LHS | DUMP_IDATA_RHS + | DUMP_IDATA_REG_SETS | DUMP_IDATA_REG_USES) + }; + +extern void dump_idata_1 (idata_t id, int); +extern void dump_idata (idata_t id); +extern void debug_idata (idata_t id); + +enum _dump_vinsn + { + DUMP_VINSN_INSN_RTX = 2, + DUMP_VINSN_TYPE = 4, + DUMP_VINSN_COUNT = 8, + DUMP_VINSN_COST = 16, + + DUMP_VINSN_ALL = (DUMP_VINSN_INSN_RTX | DUMP_VINSN_TYPE | DUMP_VINSN_COUNT + | DUMP_VINSN_COST) + }; + +extern void dump_vinsn_1 (vinsn_t, int); +extern void dump_vinsn (vinsn_t); +extern void debug_vinsn (vinsn_t); + +enum _dump_expr + { + DUMP_EXPR_VINSN = 2, + DUMP_EXPR_SPEC = 4, + DUMP_EXPR_PRIORITY = 8, + DUMP_EXPR_SCHED_TIMES = 16, + DUMP_EXPR_SPEC_DONE_DS = 32, + + DUMP_EXPR_ALL = (DUMP_EXPR_VINSN | DUMP_EXPR_SPEC | DUMP_EXPR_PRIORITY + | DUMP_EXPR_SCHED_TIMES | DUMP_EXPR_SPEC_DONE_DS) + }; + +extern void dump_expr_1 (expr_t, int); +extern void dump_expr (expr_t); +extern void debug_expr (expr_t); + /* A enumeration for dumping flags of an insn. */ -enum dump_insn_enum -{ - DUMP_INSN_UID = 1, - DUMP_INSN_BBN = 2, - DUMP_INSN_SEQNO = 4, - DUMP_INSN_PATTERN = 8, - DUMP_INSN_COUNT = 16, - DUMP_INSN_CYCLE = 32 +enum _dump_insn +{ + DUMP_INSN_ASM_P = 2, + DUMP_INSN_SCHED_NEXT = 4, + DUMP_INSN_EXPR = 8, + DUMP_INSN_AV = 16, + DUMP_INSN_SEQNO = 32, + DUMP_INSN_AFTER_STALL_P = 64, + DUMP_INSN_SCHED_CYCLE = 128, + DUMP_INSN_UID = 256, + DUMP_INSN_BBN = 512, + DUMP_INSN_PATTERN = 1024, + + DUMP_INSN_ALL = (DUMP_INSN_ASM_P | DUMP_INSN_SCHED_NEXT | DUMP_INSN_EXPR + | DUMP_INSN_AV | DUMP_INSN_SEQNO | DUMP_INSN_AFTER_STALL_P + | DUMP_INSN_SCHED_CYCLE | DUMP_INSN_UID | DUMP_INSN_BBN + | DUMP_INSN_PATTERN) }; +extern void dump_insn_1 (insn_t, int); +extern void dump_insn (insn_t); +extern void debug_insn (insn_t); extern void sel_prepare_string_for_dot_label (char *); /* When this flag is on, we are dumping to the .dot file. diff --git a/gcc/sel-sched-ir.c b/gcc/sel-sched-ir.c index 2368516881f..7e59d264c8e 100644 --- a/gcc/sel-sched-ir.c +++ b/gcc/sel-sched-ir.c @@ -53,17 +53,12 @@ #ifdef INSN_SCHEDULING #include "sel-sched-ir.h" -/* We don't have to use it except for sel_print_insn. */ -#include "sel-sched-dump.h" /* A structure used to hold various parameters of insn initialization. */ struct _insn_init insn_init; -/* Per insn data. This array is indexed by INSN_UID, but could be by INSN_LUID. */ -struct sel_insn_data *s_i_d = NULL; - -/* An array holding bb info. */ -sel_bb_info_t sel_bb_info = NULL; +/* A vector holding bb info. */ +VEC (sel_bb_info_def, heap) *sel_bb_info = NULL; /* The loop nest being pipelined. */ struct loop *current_loop_nest; @@ -75,37 +70,8 @@ static VEC(loop_p, heap) *loop_nests = NULL; /* Saves blocks already in loop regions, indexed by bb->index. */ static sbitmap bbs_in_loop_rgns = NULL; -/* A maximum UID that is used as a size to per-insn arrays. */ -int sel_max_uid = 0; - -/* For each bb header this array contains a set of live registers. - For all other insns this array has a NULLs. - - NB: All the fields, gathered in the s_i_d are initialized only for - insns that are in the current region and, therefore, can be indexed by - LUID. Except for this one. We also need to know LV sets for the - instructions, that are immediatly after the border of the region. */ -regset *lvs = NULL; - -/* Size of LVS array. */ -static int lvs_size = 0; - -/* A variable to track which part of rtx we are scanning in - sched-deps.c: sched_analyze_insn (). */ -static enum deps_where_t deps_where = DEPS_IN_NOWHERE; - -/* Is SEL_DEPS_HAS_DEP_P[DEPS_IN_X] is true, then X has a dependence. - X is from { INSN, LHS, RHS }. */ -bool sel_deps_has_dep_p[DEPS_IN_NOWHERE]; - -/* A deps context which is used to determine if two insns are dependent. */ -static struct deps _sel_deps_di, *sel_deps_di = &_sel_deps_di; - -/* Current producer and consumer insns for analyzing dependences. */ -static insn_t prod_of_dep, con_of_dep; - -/* A placeholder for saving deps info structure for switching it. */ -static struct sched_deps_info_def *saved_deps_info = NULL; +/* A vector holding data for each insn rtx. */ +VEC (sel_insn_rtx_data_def, heap) *s_i_r_d = NULL; /* This variable is used to ensure that no insns will be emitted by outer-world functions like redirect_edge_and_branch (). */ @@ -166,15 +132,11 @@ rtx exit_insn = NULL_RTX; static void fence_init (fence_t, insn_t, state_t, deps_t, void *, rtx, rtx, int, int, bool, bool); static void fence_clear (fence_t); -static int get_vinsn_type_for_insn (insn_t, bool); -static void save_deps_info (void); -static void restore_deps_info (void); -static void deps_init_id (idata_t, insn_t); +static void deps_init_id (idata_t, insn_t, bool); static void cfg_preds (basic_block, insn_t **, int *); - /* Various list functions. */ @@ -292,7 +254,8 @@ flist_clear (flist_t *lp) /* Add ORIGINAL_INSN the def list DL honoring CROSSES_CALL. */ void -def_list_add (def_list_t *dl, insn_t original_insn, bool crosses_call) +def_list_add (def_list_t *dl, insn_t original_insn, bool crosses_call, + bool needs_spec_check_p) { def_t d; _list_add (dl); @@ -300,6 +263,7 @@ def_list_add (def_list_t *dl, insn_t original_insn, bool crosses_call) d->orig_insn = original_insn; d->crosses_call = crosses_call; + d->needs_spec_check_p = needs_spec_check_p; } @@ -466,13 +430,32 @@ reset_deps_context (deps_t dc) init_deps (dc); } +static struct sched_deps_info_def _advance_deps_context_sched_deps_info = + { + NULL, + + NULL, /* start_insn */ + NULL, /* finish_insn */ + NULL, /* start_x */ + NULL, /* finish_x */ + NULL, /* start_lhs */ + NULL, /* finish_lhs */ + NULL, /* start_rhs */ + NULL, /* finish_rhs */ + haifa_note_reg_set, + haifa_note_reg_clobber, + haifa_note_reg_use, + NULL, /* note_mem_dep */ + NULL, /* note_dep */ + + 0, 0, 0 + }; + /* Process INSN and add its impact on DC. */ void advance_deps_context (deps_t dc, insn_t insn) { - insn_init.what = INSN_INIT_WHAT_INSN; - insn_init.id = INSN_ID (insn); - insn_init.todo = INSN_INIT_TODO_PREPARE_DEP; + sched_deps_info = &_advance_deps_context_sched_deps_info; deps_analyze_insn (dc, insn); } @@ -640,7 +623,7 @@ new_fences_add (flist_tail_t new_fences, insn_t insn, /* Here we should somehow choose between two DFA states. Plain reset for now. */ { - gcc_assert (bb_header_p (FENCE_INSN (f)) + gcc_assert (sel_bb_header_p (FENCE_INSN (f)) && !sched_next && !FENCE_SCHED_NEXT (f)); state_reset (FENCE_STATE (f)); @@ -809,28 +792,48 @@ free_regset_pool (void) placeholders of the insns being scheduled to allow correct update of the data sets. When update is finished, NOPs are deleted. */ +static void set_insn_init (expr_t, vinsn_t, int); +static void vinsn_attach (vinsn_t); +static void vinsn_detach (vinsn_t); + /* Emit a nop before INSN, taking it from pool. */ insn_t get_nop_from_pool (insn_t insn) { insn_t nop; + bool old_p = nop_pool.n != 0; + + if (old_p) + nop = nop_pool.v[--nop_pool.n]; + else + nop = nop_pattern; - if (nop_pool.n != 0) + insn_init.what = INSN_INIT_WHAT_INSN; + nop = emit_insn_after (nop, insn); + + if (old_p) { - nop = nop_pool.v[--nop_pool.n]; - insn_init.todo = INSN_INIT_TODO_NOTHING; + vinsn_t vi = GET_VINSN_BY_INSN (nop); + + gcc_assert (vi != NULL); + + GET_VINSN_BY_INSN (nop) = NULL; + + insn_init.todo = INSN_INIT_TODO_SSID; + set_insn_init (INSN_EXPR (insn), vi, INSN_SEQNO (insn)); } else { - nop = nop_pattern; - insn_init.todo = INSN_INIT_TODO_ALL & ~INSN_INIT_TODO_SEQNO; + insn_init.todo = INSN_INIT_TODO_LUID | INSN_INIT_TODO_SSID; + set_insn_init (INSN_EXPR (insn), NULL, INSN_SEQNO (insn)); } - insn_init.what = INSN_INIT_WHAT_INSN; - nop = emit_insn_before (nop, insn); - INSN_SEQNO (nop) = INSN_SEQNO (insn); + sel_init_new_insns (); - vinsn_attach (INSN_VI (nop)); + if (!old_p) + /* One more attach to GET_VINSN_BY_INSN to servive + sched_sel_remove_insn () in return_nop_to_pool (). */ + vinsn_attach (INSN_VINSN (nop)); return nop; } @@ -839,6 +842,11 @@ get_nop_from_pool (insn_t insn) void return_nop_to_pool (insn_t nop) { + gcc_assert (INSN_VINSN (nop) != NULL); + + GET_VINSN_BY_INSN (nop) = INSN_VINSN (nop); + + gcc_assert (INSN_IN_STREAM_P (nop)); sched_sel_remove_insn (nop); if (nop_pool.n == nop_pool.s) @@ -849,11 +857,19 @@ return_nop_to_pool (insn_t nop) } /* Free the nop pool. */ -static void +void free_nop_pool (void) { while (nop_pool.n) - vinsn_detach (INSN_VI (nop_pool.v[--nop_pool.n])); + { + insn_t nop = nop_pool.v[--nop_pool.n]; + vinsn_t vi = GET_VINSN_BY_INSN (nop); + + gcc_assert (vi != NULL && VINSN_COUNT (vi) == 1); + vinsn_detach (vi); + + GET_VINSN_BY_INSN (nop) = NULL; + } nop_pool.s = 0; free (nop_pool.v); @@ -861,18 +877,6 @@ free_nop_pool (void) } -/* Functions to work with vinsns. */ -static bool -vinsn_equal_insn_p (vinsn_t vi, insn_t insn) -{ - if (VINSN_TYPE (vi) != get_vinsn_type_for_insn (insn, VINSN_UNIQUE_P (vi))) - return false; - - return (VINSN_UNIQUE_P (vi) - ? VINSN_INSN (vi) == insn - : expr_equal_p (VINSN_PATTERN (vi), PATTERN (insn))); -} - static bool vinsn_equal_p (vinsn_t vi1, vinsn_t vi2) { @@ -881,57 +885,40 @@ vinsn_equal_p (vinsn_t vi1, vinsn_t vi2) return (VINSN_UNIQUE_P (vi1) ? VINSN_INSN (vi1) == VINSN_INSN (vi2) - : expr_equal_p (VINSN_PATTERN (vi1), VINSN_PATTERN (vi2))); + : expr_equal_p (VINSN_PATTERN (vi1), VINSN_PATTERN (vi2))); } -/* Returns whether we'll try to schedule VI's rhs and lhs separately. */ -bool -vinsn_separable_p (vinsn_t vi) +/* Returns LHS and RHS are ok to be scheduled separately. */ +static bool +lhs_and_rhs_separable_p (rtx lhs, rtx rhs) { - if (!enable_schedule_as_rhs_p) - return false; - - if (control_flow_insn_p (VINSN_INSN (vi))) - return false; - - if (VINSN_UNIQUE_P (vi)) - return false; - - if (!VINSN_LHS (vi) || !VINSN_RHS (vi)) + if (lhs == NULL || rhs == NULL) return false; /* Do not schedule CONST and CONST_INT as rhs: no point to use reg, where const can be used. Moreover, scheduling const as rhs may lead to modes mismatch cause consts don't have modes but they could be merged from branches where the same const used in different modes. */ - if (GET_CODE (VINSN_RHS (vi)) == CONST || GET_CODE (VINSN_RHS (vi)) == CONST_INT) + if (GET_CODE (lhs) == CONST || GET_CODE (rhs) == CONST_INT) return false; /* ??? Do not rename predicate registers to avoid ICEs in bundling. */ - if (COMPARISON_P (VINSN_RHS (vi))) + if (COMPARISON_P (rhs)) return false; /* Do not allow single REG to be an rhs. */ - if (REG_P (VINSN_RHS (vi))) + if (REG_P (rhs)) return false; /* See comment at find_used_regs_1 (*1) for explanation of this restriction. */ /* FIXME: remove this later. */ - if (MEM_P (VINSN_LHS (vi))) + if (MEM_P (lhs)) return false; /* This will filter all tricky things like ZERO_EXTRACT etc. For now we don't handle it. */ - if (!REG_P (VINSN_LHS (vi)) && !MEM_P (VINSN_LHS (vi))) - return false; - - /* ??? This will filter other tricky things like ZERO_EXTEND. */ - if (GET_CODE (VINSN_RHS (vi)) == ZERO_EXTEND) - return false; - - /* Do not allow renaming of cheap insns. See also PR #1. */ - if (INSN_COST (VINSN_INSN (vi)) >= 0 && INSN_COST (VINSN_INSN (vi)) < 2) + if (!REG_P (lhs) && !MEM_P (lhs)) return false; return true; @@ -939,35 +926,34 @@ vinsn_separable_p (vinsn_t vi) /* Initialize vinsn VI for INSN. Only for use from vinsn_create (). */ static void -vinsn_init (vinsn_t vi, insn_t insn, bool unique_p) +vinsn_init (vinsn_t vi, insn_t insn, bool force_unique_p) { idata_t id = xcalloc (1, sizeof (*id)); - gcc_assert (insn_init.what != INSN_INIT_WHAT_INSN - || INSN_VI (insn) == NULL); - VINSN_INSN (vi) = insn; - VINSN_ID (vi) = id; - IDATA_TYPE (id) = get_vinsn_type_for_insn (insn, unique_p); + vi->cost = -1; - IDATA_REG_SETS (id) = get_clear_regset_from_pool (); - IDATA_REG_USES (id) = get_clear_regset_from_pool (); - - if (!INSN_SIMPLEJUMP_P (insn)) - deps_init_id (id, insn); + deps_init_id (id, insn, force_unique_p); + VINSN_ID (vi) = id; - /* ??? We don't copy these fields when constructing the copy of vinsn. - This can lead us to perfomance penalties or even to infinite loop or - ICE. */ - VINSN_SCHED_CYCLE (vi) = 0; - VINSN_SCHED_TIMES (vi) = 0; + VINSN_COUNT (vi) = 0; - VINSN_SEPARABLE (vi) = vinsn_separable_p (vi); + { + int class = haifa_classify_insn (insn); + + if (class >= 2 + && (!targetm.sched.get_insn_spec_ds + || ((targetm.sched.get_insn_spec_ds (insn) & BEGIN_CONTROL) + == 0))) + VINSN_MAY_TRAP_P (vi) = true; + else + VINSN_MAY_TRAP_P (vi) = false; + } } /* Indicate that VI has become the part of an rtx object. */ -void +static void vinsn_attach (vinsn_t vi) { /* Assert that VI is not pending for deletion. */ @@ -979,12 +965,11 @@ vinsn_attach (vinsn_t vi) /* Create and init VI from the INSN. Initialize VINSN_COUNT (VI) with COUNT and use UNIQUE_P for determining the correct VINSN_TYPE (VI). */ static vinsn_t -vinsn_create (insn_t insn, int count, bool unique_p) +vinsn_create (insn_t insn, bool force_unique_p) { vinsn_t vi = xmalloc (sizeof (*vi)); - vinsn_init (vi, insn, unique_p); - VINSN_COUNT (vi) = count; + vinsn_init (vi, insn, force_unique_p); return vi; } @@ -993,8 +978,6 @@ vinsn_create (insn_t insn, int count, bool unique_p) static void vinsn_delete (vinsn_t vi) { - insn_t insn = VINSN_INSN (vi); - gcc_assert (VINSN_COUNT (vi) == 0); return_regset_to_pool (VINSN_REG_SETS (vi)); @@ -1002,10 +985,6 @@ vinsn_delete (vinsn_t vi) free (VINSN_ID (vi)); - /* ??? This should be conditional on (INSN_HAS_LUID (insn)). */ - AV_LEVEL (insn) = 0; - av_set_clear (&AV_SET (insn)); - /* This insn should not be deleted as it may have shared parts. */ /* if (!INSN_IN_STREAM_P (insn)) expr_clear (&insn); */ @@ -1014,7 +993,7 @@ vinsn_delete (vinsn_t vi) /* Indicate that VI is no longer a part of some rtx object. Remove VI if it is no longer needed. */ -void +static void vinsn_detach (vinsn_t vi) { gcc_assert (VINSN_COUNT (vi) > 0); @@ -1039,218 +1018,263 @@ vinsn_cond_branch_p (vinsn_t vi) return control_flow_insn_p (insn); } -/* This function is used after VINSN_INSN (RHS) has been altered, so - VINSN_RHS and VINSN_LHS will become valid again. */ -void -recompute_vinsn_lhs_rhs (vinsn_t vi) +/* Return latency of INSN. */ +static int +sel_insn_rtx_cost (rtx insn) +{ + int cost; + + /* A USE insn, or something else we don't need to + understand. We can't pass these directly to + result_ready_cost or insn_default_latency because it will + trigger a fatal error for unrecognizable insns. */ + if (recog_memoized (insn) < 0) + cost = 0; + else + { + cost = insn_default_latency (insn); + + if (cost < 0) + cost = 0; + } + + return cost; +} + +/* Return the cost of the VI. + !!! FIXME: Unify with haifa-sched.c: insn_cost (). */ +int +sel_vinsn_cost (vinsn_t vi) { - if (VINSN_SEPARABLE (vi)) + int cost = vi->cost; + + if (cost < 0) { - rtx pat = VINSN_PATTERN (vi); - VINSN_LHS (vi) = SET_DEST (pat); - VINSN_RHS (vi) = SET_SRC (pat); + cost = sel_insn_rtx_cost (VINSN_INSN (vi)); + vi->cost = cost; } - VINSN_SEPARABLE (vi) = vinsn_separable_p (vi); + return cost; +} + +/* Emit new insn after AFTER based on PATTERN and initialize its data from + EXPR and SEQNO. */ +insn_t +sel_gen_insn_from_rtx_after (rtx pattern, rhs_t expr, int seqno, + insn_t after) +{ + insn_t new_insn; + + insn_init.what = INSN_INIT_WHAT_INSN; + new_insn = emit_insn_after (pattern, after); + + insn_init.todo = INSN_INIT_TODO_LUID | INSN_INIT_TODO_SSID; + set_insn_init (expr, NULL, seqno); + sel_init_new_insns (); + + return new_insn; +} + +/* Emit new insn after AFTER based on EXPR and SEQNO. */ +insn_t +sel_gen_insn_from_expr_after (rhs_t expr, int seqno, insn_t after) +{ + insn_t insn = RHS_INSN (expr); + + gcc_assert (!INSN_IN_STREAM_P (insn)); + + insn_init.what = INSN_INIT_WHAT_INSN; + add_insn_after (RHS_INSN (expr), after); + + insn_init.todo = INSN_INIT_TODO_SSID; + set_insn_init (expr, EXPR_VINSN (expr), seqno); + + if (INSN_LUID (insn) == 0) + insn_init.todo |= INSN_INIT_TODO_LUID; + + sel_init_new_insns (); + + return insn; } /* Functions to work with right-hand sides. */ -/* Return whether X and Y are equal rhs'es. - Rhses should be compared as a trees, cause two rhses may become equal - through different substitutions. - Rhses with different RHS_SCHEDULE_AS_RHS values are incomparable. */ +/* Compare two vinsns as rhses if possible and as vinsns otherwise. */ bool -rhs_equal_p (rhs_t x, rhs_t y) +vinsns_correlate_as_rhses_p (vinsn_t x, vinsn_t y) { - if (RHS_SCHEDULE_AS_RHS (x) != RHS_SCHEDULE_AS_RHS (y)) - { -#ifdef ENABLE_CHECKING - /* Check for strange situations when comparing rhses like - RHS:(x+y) and WHOLE INSN:(z = x+y). */ + /* We should have checked earlier for (X == Y). */ + gcc_assert (x != y); - gcc_assert (!vinsn_equal_p (RHS_VINSN (x), RHS_VINSN (y))); -#endif - return false; - } + if (VINSN_TYPE (x) != VINSN_TYPE (y)) + return false; - if (RHS_SCHEDULE_AS_RHS (x)) + if (VINSN_SEPARABLE_P (x)) { /* Compare RHSes of VINSNs. */ - gcc_assert (VINSN_RHS (RHS_VINSN (x))); - gcc_assert (VINSN_RHS (RHS_VINSN (y))); + gcc_assert (VINSN_RHS (x)); + gcc_assert (VINSN_RHS (y)); - return expr_equal_p (VINSN_RHS (RHS_VINSN (x)), - VINSN_RHS (RHS_VINSN (y))); + return expr_equal_p (VINSN_RHS (x), + VINSN_RHS (y)); } else - { - /* Compare whole insns. */ - return vinsn_equal_p (RHS_VINSN (x), RHS_VINSN (y)); - } + /* Compare whole insns. */ + return vinsn_equal_p (x, y); } -/* Copy FROM to TO. */ -void -rhs_init (rhs_t rhs, vinsn_t vi, int spec, int priority) +/* Initialize RHS. */ +static void +init_expr (expr_t expr, vinsn_t vi, int spec, int priority, int sched_times, + ds_t spec_done_ds, ds_t spec_to_check_ds) { vinsn_attach (vi); - RHS_VINSN (rhs) = vi; - RHS_SPEC (rhs) = spec; - RHS_PRIORITY (rhs) = priority; + + EXPR_VINSN (expr) = vi; + EXPR_SPEC (expr) = spec; + EXPR_PRIORITY (expr) = priority; + EXPR_SCHED_TIMES (expr) = sched_times; + EXPR_SPEC_DONE_DS (expr) = spec_done_ds; + EXPR_SPEC_TO_CHECK_DS (expr) = spec_to_check_ds; } /* Make a copy of the rhs FROM into the rhs TO. */ void -rhs_copy (rhs_t to, rhs_t from) +copy_expr (expr_t to, expr_t from) { - rhs_init (to, RHS_VINSN (from), RHS_SPEC (from), RHS_PRIORITY (from)); + init_expr (to, EXPR_VINSN (from), EXPR_SPEC (from), EXPR_PRIORITY (from), + EXPR_SCHED_TIMES (from), EXPR_SPEC_DONE_DS (from), + EXPR_SPEC_TO_CHECK_DS (from)); } -/* Merge bits of FROM rhs to TO rhs. The two rhses should be equal. */ -static void -rhs_merge (rhs_t to, rhs_t from) +/* Merge bits of FROM rhs to TO rhs. */ +void +merge_expr_data (expr_t to, expr_t from) { - gcc_assert (rhs_equal_p (to, from)); - /* For now, we just set the spec of resulting rhs to be minimum of the specs of merged rhses. */ - if (RHS_SPEC (from) > RHS_SPEC (to)) + if (RHS_SPEC (to) > RHS_SPEC (from)) RHS_SPEC (to) = RHS_SPEC (from); - if (RHS_PRIORITY (from) > RHS_PRIORITY (to)) + if (RHS_PRIORITY (to) < RHS_PRIORITY (from)) RHS_PRIORITY (to) = RHS_PRIORITY (from); + + if (RHS_SCHED_TIMES (to) > RHS_SCHED_TIMES (from)) + RHS_SCHED_TIMES (to) = RHS_SCHED_TIMES (from); + + EXPR_SPEC_DONE_DS (to) = ds_max_merge (EXPR_SPEC_DONE_DS (to), + EXPR_SPEC_DONE_DS (from)); + + EXPR_SPEC_TO_CHECK_DS (to) |= EXPR_SPEC_TO_CHECK_DS (from); } -/* Clear the information of this RHS. */ +/* Merge bits of FROM rhs to TO rhs. Vinsns in the rhses should correlate. */ void -rhs_clear (rhs_t rhs) +merge_expr (expr_t to, expr_t from) { - vinsn_detach (RHS_VINSN (rhs)); + vinsn_t to_vi = EXPR_VINSN (to); + vinsn_t from_vi = EXPR_VINSN (from); + + gcc_assert (to_vi == from_vi + || vinsns_correlate_as_rhses_p (to_vi, from_vi)); + + merge_expr_data (to, from); } -/* Returns whether R represents INSN (or it's right-hand side, - if R has RHS_SCHEDULE_AS_RHS set to TRUE. */ -static bool -rhs_equals_insn_p (rhs_t r, insn_t insn) +/* Clear the information of this RHS. */ +void +clear_expr (rhs_t rhs) { - if (RHS_SCHEDULE_AS_RHS (r)) - { - if (VINSN_SEPARABLE (INSN_VI (insn))) - return expr_equal_p (VINSN_RHS (RHS_VINSN (r)), INSN_RHS (insn)); - else - return false; - } - else - { - /* Just compare vinsns. */ - return vinsn_equal_insn_p (RHS_VINSN (r), insn); - } + vinsn_detach (RHS_VINSN (rhs)); + RHS_VINSN (rhs) = NULL; } /* Av set functions. */ +/* Add EXPR to SETP. */ +void +av_set_add (av_set_t *setp, expr_t expr) +{ + _list_add (setp); + copy_expr (_AV_SET_EXPR (*setp), expr); +} + +/* Remove expr pointed to by IP from the av_set. */ void av_set_iter_remove (av_set_iterator *ip) { - rhs_clear (_AV_SET_RHS (*ip->lp)); + clear_expr (_AV_SET_EXPR (*ip->lp)); _list_iter_remove (ip); } -/* Search for an rhs in SET, such that it's equivalent to SOUGHT_RHS in the - sense of rhs_equal_p function. Return NULL if no such rhs is in SET was - found. */ +/* Search for an rhs in SET, such that it's equivalent to SOUGHT_VINSN in the + sense of vinsns_correlate_as_rhses_p function. Return NULL if no such rhs is + in SET was found. */ rhs_t -av_set_lookup_rhs (av_set_t set, rhs_t sought_rhs) +av_set_lookup (av_set_t set, vinsn_t sought_vinsn) { - rhs_t r; + rhs_t rhs; av_set_iterator i; - FOR_EACH_RHS (r, i, set) - if (rhs_equal_p (r, sought_rhs)) - return r; + FOR_EACH_RHS (rhs, i, set) + { + vinsn_t rhs_vinsn = RHS_VINSN (rhs); + + if (rhs_vinsn == sought_vinsn + || vinsns_correlate_as_rhses_p (rhs_vinsn, sought_vinsn)) + return rhs; + } return NULL; } -/* Search for an rhs in SET, such that it's equivalent to SOUGHT_RHS in the - sense of rhs_equal_p function, but not SOUGHT_RHS itself. - Function is used to check whether +/* Search for an rhs in SET, such that it's equivalent to SOUGHT_VINSN in the + sense of vinsns_correlate_as_rhses_p function, but not SOUGHT_VINSN itself. Returns NULL if no such rhs is in SET was found. */ rhs_t -av_set_lookup_other_equiv_rhs (av_set_t set, rhs_t sought_rhs) +av_set_lookup_other_equiv_rhs (av_set_t set, vinsn_t sought_vinsn) { - rhs_t r; + rhs_t rhs; av_set_iterator i; - FOR_EACH_RHS (r, i, set) - if (rhs_equal_p (r, sought_rhs) && r != sought_rhs) - return r; - - return NULL; -} + FOR_EACH_RHS (rhs, i, set) + { + vinsn_t rhs_vinsn = RHS_VINSN (rhs); + if (rhs_vinsn == sought_vinsn) + continue; -/* Search for rhs corresponding to INSN in SET and remove it if found, - returning TRUE. Return FALSE otherwise. */ -bool -av_set_remove_rhs_with_insn (av_set_t *set, insn_t insn) -{ - rhs_t r; - av_set_iterator i; - - FOR_EACH_RHS_1 (r, i, set) - if (rhs_equals_insn_p (r, insn)) - { - av_set_iter_remove (&i); - return true; - } + if (vinsns_correlate_as_rhses_p (rhs_vinsn, sought_vinsn)) + return rhs; + } - return false; + return NULL; } +/* Return true if there is an expr that correlates to VI in SET. */ bool -av_set_is_in_p (av_set_t set, rhs_t rhs) +av_set_is_in_p (av_set_t set, vinsn_t vi) { - return av_set_lookup_rhs (set, rhs) != NULL; -} - -rhs_t -av_set_add_vinsn (av_set_t *setp, vinsn_t vi, int spec, int priority) -{ - _list_add (setp); - rhs_init (_AV_SET_RHS (*setp), vi, spec, priority); - - return _AV_SET_RHS (*setp); + return av_set_lookup (set, vi) != NULL; } +/* Return a copy of SET. */ av_set_t -av_set_copy (av_set_t set, int flags) +av_set_copy (av_set_t set) { rhs_t rhs; av_set_iterator i; av_set_t res = NULL; - /* This flag is mandatory. */ - gcc_assert (flags & UNIQUE_RHSES); - gcc_assert (!(flags & UNIQUE_VINSNS)); - gcc_assert (!(flags & ~(UNIQUE_RHSES | UNIQUE_VINSNS))); - FOR_EACH_RHS (rhs, i, set) - { - vinsn_t vi = RHS_VINSN (rhs); - - if (flags & UNIQUE_VINSNS) - vi = vinsn_create (VINSN_INSN (vi), 0, VINSN_UNIQUE_P (vi)); - - av_set_add_vinsn (&res, vi, RHS_SPEC (rhs), RHS_PRIORITY (rhs)); - } + av_set_add (&res, rhs); return res; } -/* Makes set pointed to by TO to be the union of TO and FROM. */ +/* Makes set pointed to by TO to be the union of TO and FROM. Clear av_set + pointed to by FROMP afterwards. */ void av_set_union_and_clear (av_set_t *top, av_set_t *fromp) { @@ -1260,11 +1284,11 @@ av_set_union_and_clear (av_set_t *top, av_set_t *fromp) /* Delete from TOP all rhses, that present in FROMP. */ FOR_EACH_RHS_1 (rhs1, i, top) { - rhs_t rhs2 = av_set_lookup_rhs (*fromp, rhs1); + rhs_t rhs2 = av_set_lookup (*fromp, RHS_VINSN (rhs1)); if (rhs2) { - rhs_merge (rhs2, rhs1); + merge_expr (rhs2, rhs1); av_set_iter_remove (&i); } } @@ -1273,6 +1297,7 @@ av_set_union_and_clear (av_set_t *top, av_set_t *fromp) *i.lp = *fromp; } +/* Clear av_set pointed to by SETP. */ void av_set_clear (av_set_t *setp) { @@ -1285,12 +1310,14 @@ av_set_clear (av_set_t *setp) gcc_assert (*setp == NULL); } +/* Remove all the elements of SETP except for the first one. */ void av_set_leave_one (av_set_t *setp) { av_set_clear (&_AV_SET_NEXT (*setp)); } +/* Return the N'th element of the SET. */ rhs_t av_set_element (av_set_t set, int n) { @@ -1305,22 +1332,6 @@ av_set_element (av_set_t set, int n) return NULL; } -/* Return RHS that corresponds to INSN. - For those rhses that have RHS_SCHEDULE_AS_RHS attr set, only right-hand - sides are compared. */ -rhs_t -av_set_lookup_insn (av_set_t set, insn_t insn) -{ - rhs_t r; - av_set_iterator i; - - FOR_EACH_RHS (r, i, set) - if (rhs_equals_insn_p (r, insn)) - return r; - - return NULL; -} - /* Deletes all expressions from AVP that are conditional branches (IFs). */ void av_set_substract_cond_branches (av_set_t *avp) @@ -1333,7 +1344,8 @@ av_set_substract_cond_branches (av_set_t *avp) av_set_iter_remove (&i); } -/* Leave in AVP only those expressions, which are present in AV, and return it. */ +/* Leave in AVP only those expressions, which are present in AV, + and return it. */ void av_set_intersect (av_set_t *avp, av_set_t av) { @@ -1341,105 +1353,95 @@ av_set_intersect (av_set_t *avp, av_set_t av) rhs_t rhs; FOR_EACH_RHS_1 (rhs, i, avp) - if (av_set_lookup_rhs (av, rhs) == NULL) + if (av_set_lookup (av, RHS_VINSN (rhs)) == NULL) av_set_iter_remove (&i); } - -/* Adds INSN to av_set AVP. Helper function. - It should be used in all places where insn is added to it's own av_set. */ -void -av_set_add_insn (av_set_t *avp, insn_t insn) -{ - rhs_t added = av_set_add_vinsn (avp, INSN_VI (insn), 0, INSN_PRIORITY (insn)); - - if (enable_schedule_as_rhs_p) - { - /* If it's possible to extract rhs, set RHS_SCHEDULE_AS_RHS. - We need this to be able to lift the same rhs from different - branches and merge them. */ - if (!RHS_SCHEDULE_AS_RHS (added) - && vinsn_separable_p (RHS_VINSN (added))) - RHS_SCHEDULE_AS_RHS (added) = true; - } -} +/* Dependence hooks to initialize insn data. */ -/* Dependence hooks to initialize insn data from the dependency analyzer. */ +static struct +{ + deps_where_t where; + idata_t id; + bool force_unique_p; +} deps_init_id_data; -/* Start initializing INSN data. */ +/* Start initializing insn data. */ static void deps_init_id_start_insn (insn_t insn) { - gcc_assert (deps_where == DEPS_IN_NOWHERE); + int type; + idata_t id; - IDATA_TYPE (insn_init.id) = get_vinsn_type_for_insn (insn, false); + gcc_assert (deps_init_id_data.where == DEPS_IN_NOWHERE); - deps_where = DEPS_IN_INSN; -} + /* Determine whether INSN could be cloned and return appropriate vinsn type. + That clonable insns which can be separated into lhs and rhs have type SET. + Other clonable insns have type USE. */ + type = GET_CODE (insn); -/* Finish initializing INSN data. */ -static void -deps_init_id_finish_insn (void) -{ - gcc_assert (deps_where == DEPS_IN_INSN); + /* Only regular insns could be cloned. */ + if (type == INSN) + { + if (!deps_init_id_data.force_unique_p) + { + type = USE; + + if (enable_schedule_as_rhs_p) + type = SET; + } + } + else if (type == JUMP_INSN) + { + if (simplejump_p (insn)) + type = PC; + } + + id = deps_init_id_data.id; + + IDATA_TYPE (id) = type; + + IDATA_REG_SETS (id) = get_clear_regset_from_pool (); + IDATA_REG_USES (id) = get_clear_regset_from_pool (); - deps_where = DEPS_IN_NOWHERE; + deps_init_id_data.where = DEPS_IN_INSN; } -/* Start initializing LHS data. */ +/* Start initializing lhs data. */ static void deps_init_id_start_lhs (rtx lhs) { - gcc_assert (deps_where == DEPS_IN_INSN); + gcc_assert (deps_init_id_data.where == DEPS_IN_INSN); - gcc_assert (!IDATA_LHS (insn_init.id)); + gcc_assert (IDATA_LHS (deps_init_id_data.id) == NULL); - if (IDATA_TYPE (insn_init.id) == SET) + if (IDATA_TYPE (deps_init_id_data.id) == SET) { - IDATA_LHS (insn_init.id) = lhs; - deps_where = DEPS_IN_LHS; + IDATA_LHS (deps_init_id_data.id) = lhs; + deps_init_id_data.where = DEPS_IN_LHS; } } -/* Finish initializing LHS data. */ +/* Finish initializing lhs data. */ static void deps_init_id_finish_lhs (void) { - deps_where = DEPS_IN_INSN; + deps_init_id_data.where = DEPS_IN_INSN; } -/* Start initializing RHS data. */ +/* Downgrade to USE. */ static void -deps_init_id_start_rhs (rtx rhs) +deps_init_id_downgrade_to_use (void) { - gcc_assert (deps_where == DEPS_IN_INSN); + gcc_assert (IDATA_TYPE (deps_init_id_data.id) == SET); - /* And there was no sel_deps_reset_to_insn (). */ - if (IDATA_LHS (insn_init.id)) - { - IDATA_RHS (insn_init.id) = rhs; - deps_where = DEPS_IN_RHS; - } -} + IDATA_TYPE (deps_init_id_data.id) = USE; -/* Finish initializing RHS data. */ -static void -deps_init_id_finish_rhs (void) -{ - gcc_assert (deps_where == DEPS_IN_RHS || deps_where == DEPS_IN_INSN); + IDATA_LHS (deps_init_id_data.id) = NULL; + IDATA_RHS (deps_init_id_data.id) = NULL; - deps_where = DEPS_IN_INSN; -} - -/* Reset deps data of currently analyzing insn. */ -static void -deps_init_id_reset_deps_to_insn (void) -{ - IDATA_LHS (insn_init.id) = NULL; - IDATA_RHS (insn_init.id) = NULL; - - deps_where = DEPS_IN_INSN; + deps_init_id_data.where = DEPS_IN_INSN; } /* Note a set of REGNO. */ @@ -1448,10 +1450,11 @@ deps_init_id_note_reg_set (int regno) { haifa_note_reg_set (regno); - if (deps_where == DEPS_IN_RHS) - deps_init_id_reset_deps_to_insn (); + if (deps_init_id_data.where == DEPS_IN_RHS) + deps_init_id_downgrade_to_use (); - SET_REGNO_REG_SET (IDATA_REG_SETS (insn_init.id), regno); + if (IDATA_TYPE (deps_init_id_data.id) != PC) + SET_REGNO_REG_SET (IDATA_REG_SETS (deps_init_id_data.id), regno); } /* Note a clobber of REGNO. */ @@ -1460,10 +1463,11 @@ deps_init_id_note_reg_clobber (int regno) { haifa_note_reg_clobber (regno); - if (deps_where == DEPS_IN_RHS) - deps_init_id_reset_deps_to_insn (); + if (deps_init_id_data.where == DEPS_IN_RHS) + deps_init_id_downgrade_to_use (); - SET_REGNO_REG_SET (IDATA_REG_SETS (insn_init.id), regno); + if (IDATA_TYPE (deps_init_id_data.id) != PC) + SET_REGNO_REG_SET (IDATA_REG_SETS (deps_init_id_data.id), regno); } /* Note a use of REGNO. */ @@ -1472,25 +1476,54 @@ deps_init_id_note_reg_use (int regno) { haifa_note_reg_use (regno); - SET_REGNO_REG_SET (IDATA_REG_USES (insn_init.id), regno); + if (IDATA_TYPE (deps_init_id_data.id) != PC) + SET_REGNO_REG_SET (IDATA_REG_USES (deps_init_id_data.id), regno); } -/* Note a dependence. */ +/* Start initializing rhs data. */ static void -deps_init_id_note_mem_dep (rtx mem ATTRIBUTE_UNUSED, - rtx pending_mem ATTRIBUTE_UNUSED, - insn_t pending_insn ATTRIBUTE_UNUSED, - ds_t ds ATTRIBUTE_UNUSED) +deps_init_id_start_rhs (rtx rhs) { + gcc_assert (deps_init_id_data.where == DEPS_IN_INSN); + + /* And there was no sel_deps_reset_to_insn (). */ + if (IDATA_LHS (deps_init_id_data.id) != NULL) + { + IDATA_RHS (deps_init_id_data.id) = rhs; + deps_init_id_data.where = DEPS_IN_RHS; + } } -/* Note a memory dependence. */ +/* Finish initializing rhs data. */ static void -deps_init_id_note_dep (insn_t pro ATTRIBUTE_UNUSED, ds_t ds ATTRIBUTE_UNUSED) +deps_init_id_finish_rhs (void) { + gcc_assert (deps_init_id_data.where == DEPS_IN_RHS + || deps_init_id_data.where == DEPS_IN_INSN); + + deps_init_id_data.where = DEPS_IN_INSN; } -static struct sched_deps_info_def deps_init_id_info = +/* Finish initializing insn data. */ +static void +deps_init_id_finish_insn (void) +{ + gcc_assert (deps_init_id_data.where == DEPS_IN_INSN); + + if (IDATA_TYPE (deps_init_id_data.id) == SET) + { + rtx lhs = IDATA_LHS (deps_init_id_data.id); + rtx rhs = IDATA_RHS (deps_init_id_data.id); + + if (lhs == NULL || rhs == NULL || !lhs_and_rhs_separable_p (lhs, rhs)) + /* Downgrade to USE. */ + deps_init_id_downgrade_to_use (); + } + + deps_init_id_data.where = DEPS_IN_NOWHERE; +} + +static const struct sched_deps_info_def const_deps_init_id_sched_deps_info = { NULL, @@ -1505,359 +1538,544 @@ static struct sched_deps_info_def deps_init_id_info = deps_init_id_note_reg_set, deps_init_id_note_reg_clobber, deps_init_id_note_reg_use, - deps_init_id_note_mem_dep, - deps_init_id_note_dep, + NULL, /* note_mem_dep */ + NULL, /* note_dep */ - 0, 0, 0 + 0, /* use_cselib */ + 0, /* use_deps_list */ + 0 /* generate_spec_deps */ }; +static struct sched_deps_info_def deps_init_id_sched_deps_info; + /* Initialize instruction data for INSN in ID. */ static void -deps_init_id (idata_t id, insn_t insn) -{ - insn_init.id = id; +deps_init_id (idata_t id, insn_t insn, bool force_unique_p) +{ + struct deps _dc, *dc = &_dc; - save_deps_info (); - sched_deps_info = &deps_init_id_info; + deps_init_id_data.where = DEPS_IN_NOWHERE; + deps_init_id_data.id = id; + deps_init_id_data.force_unique_p = force_unique_p; + + init_deps (dc); - init_deps (sel_deps_di); - deps_analyze_insn (sel_deps_di, insn); - free_deps (sel_deps_di); + memcpy (&deps_init_id_sched_deps_info, + &const_deps_init_id_sched_deps_info, + sizeof (deps_init_id_sched_deps_info)); - restore_deps_info (); + if (spec_info != NULL) + deps_init_id_sched_deps_info.generate_spec_deps = 1; + + sched_deps_info = &deps_init_id_sched_deps_info; + + deps_analyze_insn (dc, insn); + + free_deps (dc); + + deps_init_id_data.id = NULL; } - + -/* Implement hooks for dependence tracking of the scheduler. */ +static bool +sel_cfg_note_p (insn_t insn) +{ + return NOTE_INSN_BASIC_BLOCK_P (insn) || LABEL_P (insn); +} -/* ??? All these macros should be plain '==', not '&'. - And we need their names to be capitalized. */ -#define deps_todo_global_init_p (insn_init.todo & INSN_INIT_TODO_GLOBAL) -#define deps_todo_prepare_dep_p (insn_init.todo & INSN_INIT_TODO_PREPARE_DEP) -#define deps_todo_has_dep_p (insn_init.todo & INSN_INIT_TODO_HAS_DEP) +/* Implement hooks for collecting fundamental insn properties like if insn is + an ASM or is within a SCHED_GROUP. */ -/* In the below hooks, we merely calculate whether or not a dependence - exists, and in what part of insn. However, we will need more data - when we'll start caching dependence requests. */ +/* Data for global dependency analysis (to initialize CANT_MOVE and + SCHED_GROUP_P). */ +static struct +{ + /* Previous insn. */ + insn_t prev_insn; +} init_global_data; -/* Start analyzing dependencies of INSN. */ +/* Determine if INSN is in the sched_group, is an asm or should not be + cloned. After that initialize its expr. */ static void -sel_start_insn (insn_t insn) +init_global_and_expr_for_insn (insn_t insn) { - gcc_assert (deps_where == DEPS_IN_NOWHERE); + if (sel_cfg_note_p (insn)) + return; + + gcc_assert (INSN_P (insn)); - if (deps_todo_has_dep_p) + if (sel_bb_header_p (insn)) + init_global_data.prev_insn = NULL_RTX; + + if (SCHED_GROUP_P (insn)) + /* Setup a sched_group. */ { - int i; + insn_t prev_insn = init_global_data.prev_insn; + + if (prev_insn) + INSN_SCHED_NEXT (prev_insn) = insn; - for (i = 0; i < DEPS_IN_NOWHERE; i++) - sel_deps_has_dep_p[i] = false; + init_global_data.prev_insn = insn; } - else if (deps_todo_global_init_p) - insn_init.global.insn = insn; + else + init_global_data.prev_insn = NULL_RTX; + + if (GET_CODE (PATTERN (insn)) == ASM_INPUT + || asm_noperands (PATTERN (insn)) >= 0) + /* Mark INSN as an asm. */ + INSN_ASM_P (insn) = true; + + { + bool force_unique_p; + ds_t spec_done_ds; + + /* Certain instructions cannot be cloned. */ + if (CANT_MOVE (insn) + || INSN_ASM_P (insn) + || SCHED_GROUP_P (insn) + || prologue_epilogue_contains (insn) + /* Exception handling insns are always unique. */ + || (flag_non_call_exceptions && can_throw_internal (insn))) + force_unique_p = true; + else + force_unique_p = false; + + if (targetm.sched.get_insn_spec_ds) + { + spec_done_ds = targetm.sched.get_insn_spec_ds (insn); + spec_done_ds = ds_get_max_dep_weak (spec_done_ds); + } + else + spec_done_ds = 0; - deps_where = DEPS_IN_INSN; + /* Initialize INSN's expr. */ + init_expr (INSN_EXPR (insn), vinsn_create (insn, force_unique_p), 0, + INSN_PRIORITY (insn), 0, spec_done_ds, 0); + } } -/* Finish analyzing dependencies of an insn. */ +static void extend_insn (void); + +/* Scan the region and initialize instruction data. */ +void +sel_init_global_and_expr (bb_vec_t bbs) +{ + { + /* ??? It would be nice to implement push / pop scheme for sched_infos. */ + const struct sched_scan_info_def ssi = + { + NULL, /* extend_bb */ + NULL, /* init_bb */ + extend_insn, /* extend_insn */ + init_global_and_expr_for_insn /* init_insn */ + }; + + sched_scan (&ssi, bbs, NULL, NULL, NULL); + } +} + +/* Perform stage 1 of finalization of the INSN's data. */ +static void +finish_global_and_expr_insn_1 (insn_t insn) +{ + if (sel_cfg_note_p (insn)) + return; + + gcc_assert (INSN_P (insn)); + + if (INSN_LUID (insn) > 0) + av_set_clear (&AV_SET (insn)); +} + +/* Perform stage 2 of finalization of the INSN's data. */ static void -sel_finish_insn (void) +finish_global_and_expr_insn_2 (insn_t insn) { - gcc_assert (deps_where == DEPS_IN_INSN); + if (sel_cfg_note_p (insn)) + return; - if (deps_todo_global_init_p) + gcc_assert (INSN_P (insn)); + + if (INSN_LUID (insn) > 0) { - insn_t insn = insn_init.global.insn; + gcc_assert (VINSN_COUNT (INSN_VINSN (insn)) == 1); - if (SCHED_GROUP_P (insn)) - { - insn_t prev_insn = insn_init.global.prev_insn; + clear_expr (INSN_EXPR (insn)); + } +} + +static void finish_insn (void); + +/* Finalize per instruction data for the whole region. */ +void +sel_finish_global_and_expr (void) +{ + { + bb_vec_t bbs; + int i; - gcc_assert (CANT_MOVE (insn)); + bbs = VEC_alloc (basic_block, heap, current_nr_blocks); - INSN_ASM_P (insn) = true; + for (i = 0; i < current_nr_blocks; i++) + VEC_quick_push (basic_block, bbs, BASIC_BLOCK (BB_TO_BLOCK (i))); - if (prev_insn) - INSN_SCHED_NEXT (prev_insn) = insn; + /* Before cleaning up insns exprs we first must clean all the cached + av_set. */ - insn_init.global.prev_insn = insn; - } - else + /* Clear INSN_AVs. */ + { + const struct sched_scan_info_def ssi = { - /* Mark possibly trapping insns. */ - if (haifa_classify_insn (insn) == TRAP_RISKY) - MAY_TRAP (insn) = 1; - - if (CANT_MOVE (insn) || prologue_epilogue_contains (insn)) - INSN_ASM_P (insn) = true; - - insn_init.global.prev_insn = NULL_RTX; - } + NULL, /* extend_bb */ + NULL, /* init_bb */ + NULL, /* extend_insn */ + finish_global_and_expr_insn_1 /* init_insn */ + }; + + sched_scan (&ssi, bbs, NULL, NULL, NULL); } - deps_where = DEPS_IN_NOWHERE; + /* Clear INSN_EXPRs. */ + { + const struct sched_scan_info_def ssi = + { + NULL, /* extend_bb */ + NULL, /* init_bb */ + NULL, /* extend_insn */ + finish_global_and_expr_insn_2 /* init_insn */ + }; + + sched_scan (&ssi, bbs, NULL, NULL, NULL); + } + + VEC_free (basic_block, heap, bbs); + } + + finish_insn (); } -/* Start analyzing dependencies of an rtx X, which is not a regular insn - or expression. */ +/* In the below hooks, we merely calculate whether or not a dependence + exists, and in what part of insn. However, we will need more data + when we'll start caching dependence requests. */ + +/* Container to hold information for dependency analysis. */ +static struct _has_dependence_data +{ + deps_t dc; + + /* A variable to track which part of rtx we are scanning in + sched-deps.c: sched_analyze_insn (). */ + deps_where_t where; + + /* Current producer. */ + insn_t pro; + + /* Current consumer. */ + vinsn_t con; + + /* Is SEL_DEPS_HAS_DEP_P[DEPS_IN_X] is true, then X has a dependence. + X is from { INSN, LHS, RHS }. */ + ds_t has_dep_p[DEPS_IN_NOWHERE]; +} has_dependence_data; + +/* Start analyzing dependencies of INSN. */ static void -sel_start_x (rtx x) +has_dependence_start_insn (insn_t insn ATTRIBUTE_UNUSED) { - if (deps_todo_global_init_p) - { - switch (GET_CODE (x)) - { - case SCRATCH: - case ASM_OPERANDS: - /* rtl.c: copy_rtx_1 () can't properly handle insns with SCRATCHes - and ASM_OPERANDS. See emit-rtl.c: copy_insn (). */ - INSN_ASM_P (insn_init.global.insn) = true; - default: - break; - } - } + gcc_assert (has_dependence_data.where == DEPS_IN_NOWHERE); + + has_dependence_data.where = DEPS_IN_INSN; } -/* Finish analyzing dependencies of an unknown rtx. */ +/* Finish analyzing dependencies of an insn. */ static void -sel_finish_x (void) +has_dependence_finish_insn (void) { + gcc_assert (has_dependence_data.where == DEPS_IN_INSN); + + has_dependence_data.where = DEPS_IN_NOWHERE; } /* Start analyzing dependencies of LHS. */ static void -sel_start_lhs (rtx lhs ATTRIBUTE_UNUSED) +has_dependence_start_lhs (rtx lhs ATTRIBUTE_UNUSED) { - gcc_assert (deps_where == DEPS_IN_INSN); + gcc_assert (has_dependence_data.where == DEPS_IN_INSN); - if (deps_todo_has_dep_p && IDATA_LHS (insn_init.id)) - deps_where = DEPS_IN_LHS; + if (VINSN_LHS (has_dependence_data.con) != NULL) + has_dependence_data.where = DEPS_IN_LHS; } /* Finish analyzing dependencies of an lhs. */ static void -sel_finish_lhs (void) +has_dependence_finish_lhs (void) { - deps_where = DEPS_IN_INSN; + has_dependence_data.where = DEPS_IN_INSN; } /* Start analyzing dependencies of RHS. */ static void -sel_start_rhs (rtx rhs ATTRIBUTE_UNUSED) +has_dependence_start_rhs (rtx rhs ATTRIBUTE_UNUSED) { - gcc_assert (deps_where == DEPS_IN_INSN); + gcc_assert (has_dependence_data.where == DEPS_IN_INSN); - if (deps_todo_has_dep_p && IDATA_RHS (insn_init.id)) - deps_where = DEPS_IN_RHS; + if (VINSN_RHS (has_dependence_data.con) != NULL) + has_dependence_data.where = DEPS_IN_RHS; } /* Start analyzing dependencies of an rhs. */ static void -sel_finish_rhs (void) +has_dependence_finish_rhs (void) { - gcc_assert (deps_where == DEPS_IN_RHS || deps_where == DEPS_IN_INSN); + gcc_assert (has_dependence_data.where == DEPS_IN_RHS + || has_dependence_data.where == DEPS_IN_INSN); - deps_where = DEPS_IN_INSN; + has_dependence_data.where = DEPS_IN_INSN; } /* Note a set of REGNO. */ static void -sel_deps_note_reg_set (int regno) +has_dependence_note_reg_set (int regno) { - if (deps_todo_prepare_dep_p) - haifa_note_reg_set (regno); - else if (deps_todo_has_dep_p) + struct deps_reg *reg_last = &has_dependence_data.dc->reg_last[regno]; + + if (!sched_insns_conditions_mutex_p (has_dependence_data.pro, + VINSN_INSN + (has_dependence_data.con))) { - struct deps_reg *reg_last = &sel_deps_di->reg_last[regno]; + ds_t *dsp = &has_dependence_data.has_dep_p[has_dependence_data.where]; + + if (reg_last->sets != NULL + || reg_last->clobbers != NULL) + *dsp = (*dsp & ~SPECULATIVE) | DEP_OUTPUT; - if ((reg_last->sets || reg_last->clobbers || reg_last->uses) - && !sched_insns_conditions_mutex_p (prod_of_dep, con_of_dep)) - sel_deps_has_dep_p[deps_where] = true; + if (reg_last->uses) + *dsp = (*dsp & ~SPECULATIVE) | DEP_ANTI; } } /* Note a clobber of REGNO. */ static void -sel_deps_note_reg_clobber (int regno) +has_dependence_note_reg_clobber (int regno) { - if (deps_todo_prepare_dep_p) - haifa_note_reg_clobber (regno); - else if (deps_todo_has_dep_p) + struct deps_reg *reg_last = &has_dependence_data.dc->reg_last[regno]; + + if (!sched_insns_conditions_mutex_p (has_dependence_data.pro, + VINSN_INSN + (has_dependence_data.con))) { - struct deps_reg *reg_last = &sel_deps_di->reg_last[regno]; + ds_t *dsp = &has_dependence_data.has_dep_p[has_dependence_data.where]; - if ((reg_last->sets || reg_last->uses) - && !sched_insns_conditions_mutex_p (prod_of_dep, con_of_dep)) - sel_deps_has_dep_p[deps_where] = true; + if (reg_last->sets) + *dsp = (*dsp & ~SPECULATIVE) | DEP_OUTPUT; + + if (reg_last->uses) + *dsp = (*dsp & ~SPECULATIVE) | DEP_ANTI; } } /* Note a use of REGNO. */ static void -sel_deps_note_reg_use (int regno) +has_dependence_note_reg_use (int regno) { - if (deps_todo_prepare_dep_p) - haifa_note_reg_use (regno); - else if (deps_todo_has_dep_p) + struct deps_reg *reg_last = &has_dependence_data.dc->reg_last[regno]; + + if (!sched_insns_conditions_mutex_p (has_dependence_data.pro, + VINSN_INSN + (has_dependence_data.con))) { - struct deps_reg *reg_last = &sel_deps_di->reg_last[regno]; + ds_t *dsp = &has_dependence_data.has_dep_p[has_dependence_data.where]; - if ((reg_last->sets || reg_last->clobbers) - && !sched_insns_conditions_mutex_p (prod_of_dep, con_of_dep)) - sel_deps_has_dep_p[deps_where] = true; + if (reg_last->sets) + *dsp = (*dsp & ~SPECULATIVE) | DEP_TRUE; + + if (reg_last->clobbers) + *dsp = (*dsp & ~SPECULATIVE) | DEP_ANTI; } } /* Note a memory dependence. */ static void -sel_note_mem_dep (rtx mem ATTRIBUTE_UNUSED, rtx pending_mem ATTRIBUTE_UNUSED, - insn_t pending_insn ATTRIBUTE_UNUSED, - ds_t ds ATTRIBUTE_UNUSED) +has_dependence_note_mem_dep (rtx mem ATTRIBUTE_UNUSED, + rtx pending_mem ATTRIBUTE_UNUSED, + insn_t pending_insn ATTRIBUTE_UNUSED, + ds_t ds ATTRIBUTE_UNUSED) { - if (deps_todo_has_dep_p - && !sched_insns_conditions_mutex_p (prod_of_dep, con_of_dep)) - sel_deps_has_dep_p[deps_where] = true; + if (!sched_insns_conditions_mutex_p (has_dependence_data.pro, + VINSN_INSN (has_dependence_data.con))) + { + ds_t *dsp = &has_dependence_data.has_dep_p[has_dependence_data.where]; + + *dsp = ds_full_merge (ds, *dsp, pending_mem, mem); + } } /* Note a dependence. */ static void -sel_note_dep (insn_t pro ATTRIBUTE_UNUSED, ds_t ds ATTRIBUTE_UNUSED) +has_dependence_note_dep (insn_t pro ATTRIBUTE_UNUSED, + ds_t ds ATTRIBUTE_UNUSED) { - if (deps_todo_has_dep_p - && !sched_insns_conditions_mutex_p (prod_of_dep, con_of_dep)) - sel_deps_has_dep_p[deps_where] = true; + if (!sched_insns_conditions_mutex_p (has_dependence_data.pro, + VINSN_INSN (has_dependence_data.con))) + { + ds_t *dsp = &has_dependence_data.has_dep_p[has_dependence_data.where]; + + *dsp = ds_full_merge (ds, *dsp, NULL_RTX, NULL_RTX); + } } -static struct sched_deps_info_def sel_sched_deps_info = +static const struct sched_deps_info_def const_has_dependence_sched_deps_info = { NULL, - sel_start_insn, - sel_finish_insn, - sel_start_x, - sel_finish_x, - sel_start_lhs, - sel_finish_lhs, - sel_start_rhs, - sel_finish_rhs, - sel_deps_note_reg_set, - sel_deps_note_reg_clobber, - sel_deps_note_reg_use, - sel_note_mem_dep, - sel_note_dep, - - 0, 0, 0 + has_dependence_start_insn, + has_dependence_finish_insn, + NULL, /* start_x */ + NULL, /* finish_x */ + has_dependence_start_lhs, + has_dependence_finish_lhs, + has_dependence_start_rhs, + has_dependence_finish_rhs, + has_dependence_note_reg_set, + has_dependence_note_reg_clobber, + has_dependence_note_reg_use, + has_dependence_note_mem_dep, + has_dependence_note_dep, + + 0, /* use_cselib */ + 0, /* use_deps_list */ + 0 /* generate_spec_deps */ }; - -/* Functions to support dependence analysis and handling. */ +static struct sched_deps_info_def has_dependence_sched_deps_info; -/* Save deps info to allow its switching. */ static void -save_deps_info (void) +setup_has_dependence_sched_deps_info (void) { - gcc_assert (saved_deps_info == NULL); - saved_deps_info = sched_deps_info; + memcpy (&has_dependence_sched_deps_info, + &const_has_dependence_sched_deps_info, + sizeof (has_dependence_sched_deps_info)); + + if (spec_info != NULL) + has_dependence_sched_deps_info.generate_spec_deps = 1; + + sched_deps_info = &has_dependence_sched_deps_info; } -/* Restore previously saved deps info structure. */ -static void -restore_deps_info (void) +void +sel_clear_has_dependence (void) { - gcc_assert (saved_deps_info != NULL); - sched_deps_info = saved_deps_info; - saved_deps_info = NULL; + int i; + + for (i = 0; i < DEPS_IN_NOWHERE; i++) + has_dependence_data.has_dep_p[i] = 0; } /* Return nonzero if RHS has is dependent upon PRED. */ -bool -has_dependence_p (rhs_t rhs, insn_t pred) +ds_t +has_dependence_p (rhs_t rhs, insn_t pred, ds_t **has_dep_pp) { - vinsn_t vi = RHS_VINSN (rhs); - insn_t insn = VINSN_INSN (vi); + struct deps _dc; int i; + ds_t ds; if (INSN_SIMPLEJUMP_P (pred)) /* Unconditional jump is just a transfer of control flow. Ignore it. */ return false; - init_deps (sel_deps_di); - /* Initialize global variables for sel_dep_* hooks. */ - prod_of_dep = pred; - con_of_dep = insn; + has_dependence_data.dc = &_dc; + init_deps (has_dependence_data.dc); /* Initialize empty dep context with information about PRED. */ - advance_deps_context (sel_deps_di, pred); + advance_deps_context (has_dependence_data.dc, pred); + + has_dependence_data.where = DEPS_IN_NOWHERE; - insn_init.id = VINSN_ID (vi); - insn_init.todo = INSN_INIT_TODO_HAS_DEP; + has_dependence_data.pro = pred; + + has_dependence_data.con = RHS_VINSN (rhs); + + sel_clear_has_dependence (); /* Now catch all dependencies that would be generated between PRED and INSN. */ - deps_analyze_insn (sel_deps_di, insn); + setup_has_dependence_sched_deps_info (); + deps_analyze_insn (has_dependence_data.dc, RHS_INSN (rhs)); + + free_deps (has_dependence_data.dc); - free_deps (sel_deps_di); + *has_dep_pp = has_dependence_data.has_dep_p; + + ds = 0; for (i = 0; i < DEPS_IN_NOWHERE; i++) - if (sel_deps_has_dep_p[i]) - return true; + ds = ds_full_merge (ds, has_dependence_data.has_dep_p[i], + NULL_RTX, NULL_RTX); - return false; + return ds; } - /* Dependence hooks implementation that checks dependence latency constraints on the insns being scheduled. The entry point for these routines is tick_check_p predicate. */ -/* An insn we are currently checking. */ -static insn_t tick_check_insn; -/* A minimal cycle for its scheduling. */ -static int tick_check_cycle; -/* Whether we have seen a true dependence while checking. */ -static int tick_check_seen_true_dep; +static struct +{ + /* An rhs we are currently checking. */ + rhs_t rhs; + + /* A minimal cycle for its scheduling. */ + int cycle; -/* Update minimal scheduling cycle for tick_check_insn given that it depends + /* Whether we have seen a true dependence while checking. */ + bool seen_true_dep_p; +} tick_check_data; + +/* Update minimal scheduling cycle for tick_check_insn given that it depends on PRO with status DS and weight DW. */ static void -tick_check_dep_with_dw (insn_t pro, ds_t ds, dw_t dw) +tick_check_dep_with_dw (insn_t pro_insn, ds_t ds, dw_t dw) { - insn_t con; - enum reg_note dt; - int tick; - - con = tick_check_insn; + rhs_t con_rhs = tick_check_data.rhs; + insn_t con_insn = RHS_INSN (con_rhs); - if (con != pro) + if (con_insn != pro_insn) { + enum reg_note dt; + int tick; + if (/* PROducer was removed from above due to pipelining. See PR8. */ - !INSN_IN_STREAM_P (pro) + !INSN_IN_STREAM_P (pro_insn) /* Or PROducer was originally on the next iteration regarding the CONsumer. */ - || (VINSN_SCHED_TIMES (INSN_VI (pro)) - - VINSN_SCHED_TIMES (INSN_VI (con))) > 1) + || (INSN_SCHED_TIMES (pro_insn) + - RHS_SCHED_TIMES (con_rhs)) > 1) /* Don't count this dependence. */ { - gcc_assert (pipelining_p); + /* ??? This assert fails on a large testcase. It is not clear + to me if the assert is right so defer debugging until a smaller + testcase is avalailable. */ + gcc_assert (1 || pipelining_p); return; } - gcc_assert (INSN_SCHED_CYCLE (pro) > 0); - dt = ds_to_dt (ds); if (dt == REG_DEP_TRUE) - tick_check_seen_true_dep = 1; + tick_check_data.seen_true_dep_p = true; - tick = INSN_SCHED_CYCLE (pro) + dep_cost (pro, dt, dw, con); + gcc_assert (INSN_SCHED_CYCLE (pro_insn) > 0); + + tick = (INSN_SCHED_CYCLE (pro_insn) + + dep_cost (pro_insn, dt, dw, con_insn)); /* When there are several kinds of dependencies between pro and con, only REG_DEP_TRUE should be taken into account. */ - if (tick > tick_check_cycle && (dt == REG_DEP_TRUE - || !tick_check_seen_true_dep)) - tick_check_cycle = tick; + if (tick > tick_check_data.cycle + && (dt == REG_DEP_TRUE || !tick_check_data.seen_true_dep_p)) + tick_check_data.cycle = tick; } } @@ -1881,7 +2099,7 @@ tick_check_note_mem_dep (rtx mem1, rtx mem2, insn_t pro, ds_t ds) tick_check_dep_with_dw (pro, ds, dw); } -static struct sched_deps_info_def tick_check_deps_info = +static struct sched_deps_info_def _tick_check_sched_deps_info = { NULL, @@ -1905,88 +2123,59 @@ static struct sched_deps_info_def tick_check_deps_info = /* Returns true when VI's insn can be scheduled on the current cycle of FENCE. That is, all data from possible producers in DC_ORIG is ready. */ bool -tick_check_p (vinsn_t vi, deps_t dc_orig, fence_t fence) +tick_check_p (rhs_t rhs, deps_t dc_orig, fence_t fence) { struct deps _dc, *dc = &_dc; /* Initialize variables. */ - tick_check_insn = VINSN_INSN (vi); - tick_check_cycle = 0; - tick_check_seen_true_dep = 0; - - /* Switch hooks. */ - save_deps_info (); - sched_deps_info = &tick_check_deps_info; + tick_check_data.rhs = rhs; + tick_check_data.cycle = 0; + tick_check_data.seen_true_dep_p = false; /* Calculate TICK_CHECK_CYCLE. */ copy_deps_context (dc, dc_orig); - deps_analyze_insn (dc, tick_check_insn); - free_deps (dc); - /* Restore hooks. */ - restore_deps_info (); + sched_deps_info = &_tick_check_sched_deps_info; + deps_analyze_insn (dc, RHS_INSN (rhs)); - return FENCE_CYCLE (fence) >= tick_check_cycle; + free_deps (dc); + + return FENCE_CYCLE (fence) >= tick_check_data.cycle; } /* Functions to work with insns. */ -/* Returns true if LHS of INSN is a register and it's the same register as R2. */ +/* Returns true if LHS of INSN is a register and it's the same register as + R2. */ bool -lhs_equals_reg_p (insn_t insn, rtx reg) +lhs_of_insn_equals_to_reg_p (insn_t insn, rtx reg) { rtx lhs = INSN_LHS (insn); - - if (!lhs || !reg) - return false; - return REG_P (lhs) && (REGNO (lhs) == REGNO (reg)); -} -/* Determine whether INSN could be cloned and return appropriate vinsn type. - All clonable VINSNS have type SET. When UNIQUE_P is true, always require - a unique vinsn. */ -static int -get_vinsn_type_for_insn (insn_t insn, bool unique_p) -{ - int code = GET_CODE (insn), unique_vinsn_code = -1; + gcc_assert (reg != NULL_RTX); - /* If UNIQUE_P, the insn is considered unique in any case. */ - if (unique_p) - return unique_vinsn_code; - /* Only regular insns could be cloned. */ - if (code != INSN) - return unique_vinsn_code; - /* ASM and exception handling insns are always unique. */ - if (INSN_ASM_P (insn) - || (flag_non_call_exceptions && can_throw_internal (insn))) - return unique_vinsn_code; - - /* Everything else is considered clonable and handled like SET. */ - return SET; -} - -/* True when INSN is a basic block header. */ -bool -bb_header_p (insn_t insn) -{ - gcc_assert (insn && INSN_P (insn)); + if (lhs == NULL) + return false; - return insn == exit_insn || NOTE_INSN_BASIC_BLOCK_P (PREV_INSN (insn)); + return REG_P (lhs) && (REGNO (lhs) == REGNO (reg)); } -/* Returns whether insn is valid in terms of target architecture. */ +/* Returns whether INSN_RTX is valid in terms of target architecture. + Don't use this function inside gcc_assert () because it has side effects: + e.g. it initializes INSN_CODE (INSN_RTX). */ bool -insn_valid_p (insn_t insn) +insn_rtx_valid (rtx insn_rtx) { /* Reset the INSN_CODE. After register replacement it might have became a different insn. */ - INSN_CODE (insn) = -1; - if (recog_memoized (insn) >= 0) + INSN_CODE (insn_rtx) = -1; + + if (recog_memoized (insn_rtx) >= 0) { - extract_insn (insn); - return constrain_operands (reload_completed) ? true : false; + extract_insn (insn_rtx); + return constrain_operands (reload_completed) != 0; } else return false; @@ -2018,66 +2207,6 @@ insn_eligible_for_subst_p (insn_t insn) return false; } -/* An implementation of RTL_HOOKS_INSN_ADDED hook. The hook is used for - initializing per-insn data structures when new insn is emitted. */ -static void -sel_rtl_insn_added (insn_t insn) -{ - gcc_assert (can_add_insns_p); - - if (/* We have added an instruction to the instruction stream. */ - insn_init.what == INSN_INIT_WHAT_INSN) - { - if (insn_init.how == INSN_INIT_HOW_NOW) - { - - if (INSN_P (insn) && BLOCK_FOR_INSN (insn)) - gcc_assert (CONTAINING_RGN (BB_TO_BLOCK (0)) - == CONTAINING_RGN (BLOCK_FOR_INSN (insn)->index)); - - if ((insn_init.todo & INSN_INIT_TODO_LUID) - && INSN_NEED_LUID_P (insn)) - sched_data_update (NULL, NULL, insn); - } - else - /* Initialize a bit later because something (e.g. CFG) is not - consistent yet. This clause is used to handle insntruction that are - being added by CFG manipulation routines. These insns will be - initialized when sel_insn_deferred_init () is called. */ - { - struct insn_init_how_deferred_def *p = &insn_init.how_deferred; - - gcc_assert (insn_init.how == INSN_INIT_HOW_DEFERRED); - - if (p->n == p->size) - p->insns = xrealloc (p->insns, ((p->size = 2 * p->size + 1) - * sizeof (*p->insns))); - - p->insns[p->n++] = insn; - } - } -} - -/* Perform deferred initialization of insns. This is used to process - a new jump that may be created by redirect_edge. */ -static void -sel_insn_deferred_init (void) -{ - int i; - struct insn_init_how_deferred_def *p = &insn_init.how_deferred; - - gcc_assert (insn_init.how == INSN_INIT_HOW_DEFERRED); - - insn_init.how = INSN_INIT_HOW_NOW; - - /* NB: This can be optimized to a single call to - sched_data_update (NULL, NULL, NULL, p->insns) . */ - for (i = 0; i < p->n; i++) - sel_rtl_insn_added (p->insns[i]); - - p->n = 0; -} - /* Extracts machine mode MODE and destination location DST_LOC for given INSN. */ void @@ -2117,50 +2246,6 @@ bookkeeping_can_be_created_if_moved_through_p (insn_t jump) return false; } -/* Makes a copy of INSN with emit_insn call and returns it. */ -insn_t -copy_insn_out_of_stream (vinsn_t vi) -{ - insn_t new_insn; - insn_t insn = PATTERN (VINSN_INSN (vi)); - - insn_init.what = INSN_INIT_WHAT_INSN; - insn_init.todo = INSN_INIT_TODO_LUID | INSN_INIT_TODO_INIT_ID; - start_sequence (); - new_insn = emit_insn (expr_copy (insn)); - end_sequence (); - - /* Fill the only field in s_i_d to make vinsn_create work. */ - INSN_TYPE (new_insn) = VINSN_TYPE (vi); - - /* Copy other fields. */ - INSN_COST (new_insn) = INSN_COST (VINSN_INSN (vi)); - INSN_PRIORITY (new_insn) = INSN_PRIORITY (VINSN_INSN (vi)); - /* !!! Initialize INSN_SCHED_TIMES () and possibly other field. */ - - return new_insn; -} - -/* Inserts a copy of INSN_TO_COPY before the insn INS_BEFORE via the call of - emit_insn_before. - FIXME: is expr_copy necessary? */ -insn_t -copy_insn_and_insert_before (insn_t insn_to_copy, insn_t ins_before) -{ - insn_t new_insn; - - insn_init.what = INSN_INIT_WHAT_INSN; - insn_init.todo = INSN_INIT_TODO_LUID | INSN_INIT_TODO_INIT_ID; - - new_insn = emit_insn_before (expr_copy (PATTERN (insn_to_copy)), - ins_before); - - /* Fill the only field in s_i_d to make vinsn_create work. */ - INSN_TYPE (new_insn) = INSN_TYPE (insn_to_copy); - - return new_insn; -} - /* Rip-off INSN from the insn stream. */ void sched_sel_remove_insn (insn_t insn) @@ -2175,7 +2260,7 @@ sched_sel_remove_insn (insn_t insn) PREV_INSN (insn) = NULL_RTX; NEXT_INSN (insn) = NULL_RTX; - vinsn_detach (INSN_VI (insn)); + clear_expr (INSN_EXPR (insn)); } /* Transfer av and lv sets from FROM to TO. */ @@ -2199,7 +2284,7 @@ transfer_data_sets (insn_t to, insn_t from) /* Estimate number of the insns in BB. */ static int -sched_sel_estimate_number_of_insns (basic_block bb) +sel_estimate_number_of_insns (basic_block bb) { int res = 0; insn_t insn = NEXT_INSN (BB_HEAD (bb)), next_tail = NEXT_INSN (BB_END (bb)); @@ -2213,181 +2298,312 @@ sched_sel_estimate_number_of_insns (basic_block bb) /* We don't need separate luids for notes or labels. */ static int -sched_sel_which_luid (insn_t note) +sel_luid_for_non_insn (rtx x) { - gcc_assert (NOTE_P (note) || LABEL_P (note)); + gcc_assert (NOTE_P (x) || LABEL_P (x)); + return -1; } -/* Extend the per-insn data structures. */ -static void -sel_extend_insn_info (void) +/* Extend data structures that are indexed by INSN_UID. */ +void +sel_extend_insn_rtx_data (void) { - int lvs_size_1 = get_max_uid () + 1; + sched_extend_target (); + sched_deps_local_init (false); - lvs = xrecalloc (lvs, lvs_size_1, lvs_size, sizeof (*lvs)); - lvs_size = lvs_size_1; + { + int new_size = get_max_uid () + 1; - s_i_d = xrecalloc (s_i_d, sched_max_uid, sel_max_uid, sizeof (*s_i_d)); - sel_max_uid = sched_max_uid; + VEC_safe_grow_cleared (sel_insn_rtx_data_def, heap, s_i_r_d, new_size); + } } -/* Init data for INSN. */ -static void -sel_init_insn_info (insn_t insn) +/* Finalize data structures that are indexed by INSN_UID. */ +void +sel_finish_insn_rtx_data (void) { - int todo = insn_init.todo; + sched_deps_local_finish (); + VEC_free (sel_insn_rtx_data_def, heap, s_i_r_d); - if (INSN_P (insn) && todo) - { - INSN_COST (insn) = insn_cost (insn, 0, 0); - } + /* Target will finalize its data structures in + targetm.sched.md_global_finish (). */ +} - if (INSN_P (insn)) - gcc_assert (can_add_real_insns_p); +/* Return seqno of the only predecessor of INSN. */ +static int +get_seqno_of_a_pred (insn_t insn) +{ + int seqno; - if (INSN_P (insn) && (todo & INSN_INIT_TODO_GLOBAL)) + gcc_assert (INSN_SIMPLEJUMP_P (insn)); + + if (!sel_bb_header_p (insn)) + seqno = INSN_SEQNO (PREV_INSN (insn)); + else { - if (bb_header_p (insn)) + basic_block bb = BLOCK_FOR_INSN (insn); + + if (single_pred_p (bb) + && !in_current_region_p (single_pred (bb))) + /* We can have preds outside a region when splitting edges + for pipelining of an outer loop. Use succ instead. */ { - gcc_assert (!CANT_MOVE (insn)); + insn_t succ; - insn_init.global.prev_insn = NULL_RTX; + gcc_assert (flag_sel_sched_pipelining_outer_loops + && current_loop_nest); - init_deps (&insn_init.global.deps); - deps_start_bb (&insn_init.global.deps, insn); - } + succ = cfg_succ_1 (insn, (SUCCS_NORMAL | SUCCS_SKIP_TO_LOOP_EXITS)); + gcc_assert (succ != NULL); - insn_init.todo = INSN_INIT_TODO_GLOBAL; + seqno = INSN_SEQNO (succ); + } + else + { + insn_t *preds; + int n; - /* This will initialize CANT_MOVE () and SCHED_GROUP_P () data. */ - deps_analyze_insn (&insn_init.global.deps, insn); + cfg_preds (BLOCK_FOR_INSN (insn), &preds, &n); + gcc_assert (n == 1); - if (BB_END (BLOCK_FOR_INSN (insn)) == insn) - free_deps (&insn_init.global.deps); + seqno = INSN_SEQNO (preds[0]); + + free (preds); + } } - if (INSN_P (insn) && (todo & INSN_INIT_TODO_VINSN)) - { - gcc_assert (insn_init.what == INSN_INIT_WHAT_INSN); - - INSN_SIMPLEJUMP_P (insn) = !!simplejump_p (insn); +#ifdef ENABLE_CHECKING + { + insn_t succ = cfg_succ (insn); - INSN_VI (insn) = vinsn_create (insn, 1, false); - } + gcc_assert ((succ != NULL && seqno <= INSN_SEQNO (succ)) + || (succ == NULL && flag_sel_sched_pipelining_outer_loops)); + } +#endif - if (INSN_P (insn) && (todo & INSN_INIT_TODO_LV_SET)) - /* This is used to initialize spurious jumps generated by - sel_split_block () / sel_redirect_edge (). */ - { - rtx succ = cfg_succ_1 (insn, SUCCS_ALL); + return seqno; +} - gcc_assert (insn_init.what == INSN_INIT_WHAT_INSN - && INSN_SIMPLEJUMP_P (insn) - && !LV_SET_VALID_P (insn)); +/* Data for each insn in current region. */ +VEC (sel_insn_data_def, heap) *s_i_d = NULL; - if (bb_header_p (insn)) - { - LV_SET (insn) = get_regset_from_pool (); - COPY_REG_SET (LV_SET (insn), LV_SET (succ)); - } +/* Extend data structures for insns from current region. */ +static void +extend_insn (void) +{ + /* Extend data structures that are indexed by INSN_UID. */ + sel_extend_insn_rtx_data (); + + /* Extend data structures for insns from current region. */ + VEC_safe_grow_cleared (sel_insn_data_def, heap, s_i_d, + sched_max_luid); +} + +/* Finalize data structures for insns from current region. */ +static void +finish_insn (void) +{ + VEC_free (sel_insn_data_def, heap, s_i_d); + deps_finish_d_i_d (); +} + +static insn_vec_t new_insns = NULL; + +/* An implementation of RTL_HOOKS_INSN_ADDED hook. The hook is used for + initializing data structures when new insn is emitted. + This hook remembers all relevant instuctions which can be initialized later + with the call to sel_init_new_insns (). */ +static void +sel_rtl_insn_added (insn_t insn) +{ + gcc_assert (can_add_insns_p + && (!INSN_P (insn) || can_add_real_insns_p)); + + if (!INSN_P (insn) || insn_init.what == INSN_INIT_WHAT_INSN_RTX) + return; + + gcc_assert (BLOCK_FOR_INSN (insn) == NULL + || (VEC_length (sel_bb_info_def, sel_bb_info) + <= (unsigned) BLOCK_NUM (insn)) + || (CONTAINING_RGN (BB_TO_BLOCK (0)) + == CONTAINING_RGN (BLOCK_NUM (insn)))); + + /* Initialize a bit later because something (e.g. CFG) is not + consistent yet. These insns will be initialized when + sel_init_new_insns () is called. */ + VEC_safe_push (rtx, heap, new_insns, insn); +} + +/* A proxy to pass initialization data to init_insn (). */ +static sel_insn_data_def _insn_init_ssid; +static sel_insn_data_t insn_init_ssid = &_insn_init_ssid; + +/* A dummy variable used in set_insn_init () and init_insn (). */ +static vinsn_t empty_vinsn = NULL; + +/* Set all necessary data for initialization of the new insn[s]. */ +static void +set_insn_init (expr_t expr, vinsn_t vi, int seqno) +{ + expr_t x = &insn_init_ssid->_expr; + + copy_expr (x, expr); + + if (vi != NULL) + change_vinsn_in_expr (x, vi); + else + change_vinsn_in_expr (x, empty_vinsn); + + insn_init_ssid->seqno = seqno; +} + +/* Init data for INSN. */ +static void +init_insn (insn_t insn) +{ + expr_t expr; + expr_t x; + sel_insn_data_t ssid = insn_init_ssid; + + /* The fields mentioned below are special and hence are not being + propagated to the new insns. */ + gcc_assert (!ssid->asm_p && ssid->sched_next == NULL + && ssid->av_level == 0 && ssid->av == NULL + && !ssid->after_stall_p && ssid->sched_cycle == 0); + + gcc_assert (INSN_P (insn) && INSN_LUID (insn) > 0); + + expr = INSN_EXPR (insn); + x = &ssid->_expr; + + copy_expr (expr, x); + + if (EXPR_VINSN (x) == empty_vinsn) + change_vinsn_in_expr (expr, vinsn_create (insn, false)); + + INSN_SEQNO (insn) = ssid->seqno; +} + +/* This is used to initialize spurious jumps generated by + sel_split_block () / sel_redirect_edge (). */ +static void +init_simplejump (insn_t insn) +{ + rtx succ = cfg_succ_1 (insn, SUCCS_ALL); + + gcc_assert (LV_SET (insn) == NULL); + + if (sel_bb_header_p (insn)) + { + LV_SET (insn) = get_regset_from_pool (); + COPY_REG_SET (LV_SET (insn), LV_SET (succ)); } - if (INSN_P (insn) && (todo & INSN_INIT_TODO_SEQNO)) + init_expr (INSN_EXPR (insn), vinsn_create (insn, false), 0, 0, 0, 0, 0); + + INSN_SEQNO (insn) = get_seqno_of_a_pred (insn); +} + +/* This is used to move lv_sets to the first insn of basic block if that + insn was emitted by the target. */ +static void +insn_init_move_lv_set_if_bb_header (insn_t insn) +{ + if (sel_bb_header_p (insn)) { - int seqno; - insn_t succ; + insn_t next = NEXT_INSN (insn); - gcc_assert (INSN_SIMPLEJUMP_P (insn)); + gcc_assert (INSN_LUID (insn) == 0); - if (!bb_header_p (insn)) - seqno = INSN_SEQNO (PREV_INSN (insn)); - else + /* Find the insn that used to be a bb_header. */ + while (INSN_LUID (next) == 0) { - insn_t *preds; - int i, n; - basic_block bb = BLOCK_FOR_INSN (insn); + gcc_assert (!sel_bb_end_p (next)); + next = NEXT_INSN (next); + } - /* We can have preds outside a region when splitting edges - for pipelining of an outer loop. Use succ instead. */ - if (single_pred_p (bb) - && !in_current_region_p (single_pred (bb))) - { - - gcc_assert (flag_sel_sched_pipelining_outer_loops - && current_loop_nest); + gcc_assert (LV_SET_VALID_P (next)); - succ = cfg_succ_1 (insn, (SUCCS_NORMAL - | SUCCS_SKIP_TO_LOOP_EXITS)); - gcc_assert (succ); + LV_SET (insn) = LV_SET (next); + LV_SET (next) = NULL; + } +} - seqno = INSN_SEQNO (succ); - } - else - { - - cfg_preds (BLOCK_FOR_INSN (insn), &preds, &n); - seqno = INSN_SEQNO (preds[0]); - - for (i = 1; i < n; i++) - { - gcc_unreachable (); - if (INSN_SEQNO (preds[i]) > seqno) - seqno = INSN_SEQNO (preds[i]); - } - - free (preds); - } - } +/* Perform deferred initialization of insns. This is used to process + a new jump that may be created by redirect_edge. */ +void +sel_init_new_insns (void) +{ + int todo = insn_init.todo; + + if (todo & INSN_INIT_TODO_LUID) + sched_init_luids (NULL, NULL, new_insns, NULL); + + if (todo & INSN_INIT_TODO_SSID) + { + const struct sched_scan_info_def ssi = + { + NULL, /* extend_bb */ + NULL, /* init_bb */ + extend_insn, /* extend_insn */ + init_insn /* init_insn */ + }; - succ = cfg_succ (insn); - gcc_assert ((succ && seqno <= INSN_SEQNO (succ)) - || (!succ && flag_sel_sched_pipelining_outer_loops)); + sched_scan (&ssi, NULL, NULL, new_insns, NULL); - INSN_SEQNO (insn) = seqno; + clear_expr (&insn_init_ssid->_expr); } - if (INSN_P (insn) && (todo & INSN_INIT_TODO_MOVE_LV_SET_IF_BB_HEADER)) + if (todo & INSN_INIT_TODO_SIMPLEJUMP) { - if (bb_header_p (insn)) + const struct sched_scan_info_def ssi = { - insn_t next = NEXT_INSN (insn); + NULL, /* extend_bb */ + NULL, /* init_bb */ + extend_insn, /* extend_insn */ + init_simplejump /* init_insn */ + }; - gcc_assert (BLOCK_FOR_INSN (next) == BLOCK_FOR_INSN (insn) - && LV_SET_VALID_P (next)); + sched_scan (&ssi, NULL, NULL, new_insns, NULL); + } + + if (todo & INSN_INIT_TODO_MOVE_LV_SET_IF_BB_HEADER) + { + const struct sched_scan_info_def ssi = + { + NULL, /* extend_bb */ + NULL, /* init_bb */ + sel_extend_insn_rtx_data, /* extend_insn */ + insn_init_move_lv_set_if_bb_header /* init_insn */ + }; - LV_SET (insn) = LV_SET (next); - LV_SET (next) = NULL; - } + sched_scan (&ssi, NULL, NULL, new_insns, NULL); } - insn_init.todo = todo; + VEC_truncate (rtx, new_insns, 0); } -/* Free the per-insn data structures. */ -static void -sel_finish_insn_info (void) +/* Finalize new_insns data. */ +void +sel_finish_new_insns (void) { - free_nop_pool (); + gcc_assert (VEC_empty (rtx, new_insns)); - free (s_i_d); - s_i_d = NULL; - sel_max_uid = 0; + VEC_free (rtx, heap, new_insns); } -/* Return the cost of INSN as estimated by DFA. This function properly handles - ASMs, USEs etc. */ +/* Return the cost of VINSN as estimated by DFA. This function properly + handles ASMs, USEs etc. */ int -dfa_cost (insn_t insn, fence_t fence) +vinsn_dfa_cost (vinsn_t vinsn, fence_t fence) { + rtx insn = VINSN_INSN (vinsn); + if (recog_memoized (insn) < 0) { - bool asm_p; - - asm_p = (GET_CODE (PATTERN (insn)) == ASM_INPUT - || asm_noperands (PATTERN (insn)) >= 0); - - if (!FENCE_STARTS_CYCLE_P (fence) && asm_p) + if (!FENCE_STARTS_CYCLE_P (fence) && VINSN_UNIQUE_P (vinsn) + && INSN_ASM_P (insn)) /* This is asm insn which is tryed to be issued on the cycle not first. Issue it on the next cycle. */ return 1; @@ -2429,14 +2645,12 @@ init_lv_set_for_insn (insn_t insn, basic_block bb) COPY_REG_SET (LV_SET (insn), glat_start[bb->index]); } +/* Initialize lv set of all bb headers. */ void init_lv_sets (void) { basic_block bb; - lvs_size = get_max_uid () + 1; - lvs = xcalloc (lvs_size, sizeof (*lvs)); - /* Initialization of the LV sets. */ FOR_EACH_BB (bb) { @@ -2454,108 +2668,229 @@ init_lv_sets (void) init_lv_set_for_insn (exit_insn, EXIT_BLOCK_PTR); } -void +/* Release lv set of HEAD. */ +static void release_lv_set_for_insn (rtx head) { int uid = INSN_UID (head); - if (uid < lvs_size && lvs[uid] != NULL) - return_regset_to_pool (lvs[uid]); -} + if (((unsigned) uid) < VEC_length (sel_insn_rtx_data_def, s_i_r_d)) + { + regset lv = LV_SET (head); + if (lv != NULL) + { + return_regset_to_pool (lv); + LV_SET (head) = NULL; + } + } +} +/* Finalize lv sets of all bb headers. */ void free_lv_sets (void) { - free (lvs); - lvs = NULL; - lvs_size = 0; + basic_block bb; + + gcc_assert (LV_SET_VALID_P (exit_insn)); + release_lv_set_for_insn (exit_insn); + + /* !!! FIXME: Walk through bb_headers only. */ + FOR_EACH_BB (bb) + { + insn_t head; + insn_t next_tail; + + get_ebb_head_tail (bb, bb, &head, &next_tail); + next_tail = NEXT_INSN (next_tail); + + /* We should scan through all the insns because bundling could + have emitted new insns at the bb headers. */ + while (head != next_tail) + { + release_lv_set_for_insn (head); + head = NEXT_INSN (head); + } + } } -/* Functions to work with control-flow graph. */ +/* Variables to work with control-flow graph. */ /* The basic block that already has been processed by the sched_data_update (), but hasn't been in sel_add_or_remove_bb () yet. */ static VEC (basic_block, heap) *last_added_blocks = NULL; -/* Return first insn in BB. BB must not be empty. - ??? BB_HEAD must be used instead of this. */ +/* Functions to work with control-flow graph. */ + +/* Return the first real insn of BB. If STRICT_P is true, then assume + that BB is current region and hence has no unrelevant notes in it. */ +static insn_t +sel_bb_header_1 (basic_block bb, bool strict_p) +{ + insn_t header; + + if (bb == EXIT_BLOCK_PTR) + { + gcc_assert (exit_insn != NULL_RTX); + header = exit_insn; + } + else + { + if (strict_p) + { + rtx note = bb_note (bb); + + if (note != BB_END (bb)) + header = NEXT_INSN (note); + else + header = NULL_RTX; + } + else + { + rtx head, tail; + + get_ebb_head_tail (bb, bb, &head, &tail); + + if (INSN_P (head)) + header = head; + else + header = NULL_RTX; + } + } + + return header; +} + +/* Return the first real insn of BB. */ insn_t -bb_head (basic_block bb) +sel_bb_header (basic_block bb) { - insn_t head; - insn_t tail; + insn_t header = sel_bb_header_1 (bb, true); - get_ebb_head_tail (bb, bb, &head, &tail); + gcc_assert (header == NULL_RTX || INSN_P (header)); - return INSN_P (head) ? head : NULL_RTX; + return header; } -/* True when BB belongs to the current scheduling region. */ +/* Return true if INSN is a basic block header. */ bool -in_current_region_p (basic_block bb) +sel_bb_header_p (insn_t insn) { - if (bb->index < NUM_FIXED_BLOCKS) - return false; + gcc_assert (insn != NULL_RTX && INSN_P (insn)); - return CONTAINING_RGN (bb->index) == CONTAINING_RGN (BB_TO_BLOCK (0)); + return insn == sel_bb_header (BLOCK_FOR_INSN (insn)); } -/* An implementation of create_basic_block hook, which additionally updates - per-bb data structures. */ -basic_block -sel_create_basic_block (void *headp, void *endp, basic_block after) +/* Return true if BB has no real insns. If STRICT_P is true, then assume + that BB is current region and hence has no unrelevant notes in it. */ +bool +sel_bb_empty_p_1 (basic_block bb, bool strict_p) { - basic_block new_bb; - - gcc_assert (flag_sel_sched_pipelining_outer_loops - || last_added_blocks == NULL); + return sel_bb_header_1 (bb, strict_p) == NULL_RTX; +} - new_bb = old_create_basic_block (headp, endp, after); - VEC_safe_push (basic_block, heap, last_added_blocks, new_bb); +/* Return true if BB has no real insns. If STRICT_P is true, then assume + that BB is current region and hence has no unrelevant notes in it. */ +bool +sel_bb_empty_p (basic_block bb) +{ + return sel_bb_empty_p_1 (bb, true); +} - sched_data_update (NULL, new_bb, NULL); +/* Return last insn of BB. */ +insn_t +sel_bb_end (basic_block bb) +{ + gcc_assert (!sel_bb_empty_p (bb)); - return new_bb; + return BB_END (bb); } -/* True when BB is empty. A block containing only a label is considered - empty as well. */ +/* Return true if INSN is the last insn in its basic block. */ bool -bb_empty_p (basic_block bb) +sel_bb_end_p (insn_t insn) { - insn_t head, tail; + return insn == sel_bb_end (BLOCK_FOR_INSN (insn)); +} - if (bb == EXIT_BLOCK_PTR) +/* True when BB belongs to the current scheduling region. */ +bool +in_current_region_p (basic_block bb) +{ + if (bb->index < NUM_FIXED_BLOCKS) return false; - get_ebb_head_tail (bb, bb, &head, &tail); - return head == tail && NOTE_P (head); + return CONTAINING_RGN (bb->index) == CONTAINING_RGN (BB_TO_BLOCK (0)); } /* Extend per bb data structures. */ static void -sel_extend_bb_info (void) +extend_bb (void) { - sel_bb_info = xrecalloc (sel_bb_info, last_basic_block, - sched_last_basic_block, sizeof (*sel_bb_info)); + VEC_safe_grow_cleared (sel_bb_info_def, heap, sel_bb_info, last_basic_block); } -/* Free per-bb data structures. */ +/* Remove all notes from BB. */ static void -sel_finish_bb_info (void) +init_bb (basic_block bb) +{ + remove_notes (bb_note (bb), BB_END (bb)); + BB_NOTE_LIST (bb) = note_list; +} + +void +sel_init_bbs (bb_vec_t bbs, basic_block bb) { - free (sel_bb_info); - sel_bb_info = NULL; + const struct sched_scan_info_def ssi = + { + extend_bb, /* extend_bb */ + init_bb, /* init_bb */ + NULL, /* extend_insn */ + NULL /* init_insn */ + }; + + sched_scan (&ssi, bbs, bb, new_insns, NULL); } -/* Remove all notes from BB. */ +/* Restore other notes for the whole region. */ static void -sel_remove_notes (basic_block bb) +sel_restore_other_notes (void) { - remove_notes (bb_note (bb), BB_END (bb)); - BB_NOTE_LIST (bb) = note_list; + int bb; + + for (bb = 0; bb < current_nr_blocks; bb++) + { + basic_block first, last; + + first = EBB_FIRST_BB (bb); + last = EBB_LAST_BB (bb)->next_bb; + + do + { + note_list = BB_NOTE_LIST (first); + restore_other_notes (NULL, first); + BB_NOTE_LIST (first) = NULL_RTX; + + first = first->next_bb; + } + while (first != last); + } +} + +static void sel_remove_loop_preheader (void); + +/* Free per-bb data structures. */ +void +sel_finish_bbs (void) +{ + sel_restore_other_notes (); + + /* Remove current loop preheader from this loop. */ + if (flag_sel_sched_pipelining_outer_loops && current_loop_nest) + sel_remove_loop_preheader (); + + VEC_free (sel_bb_info_def, heap, sel_bb_info); } /* Return a number of INSN's successors honoring FLAGS. */ @@ -2572,6 +2907,25 @@ cfg_succs_n (insn_t insn, int flags) return n; } +/* Return true if INSN has a single successor of type FLAGS. */ +bool +sel_insn_has_single_succ_p (insn_t insn, int flags) +{ + insn_t succ; + succ_iterator si; + bool first_p = true; + + FOR_EACH_SUCC_1 (succ, si, insn, flags) + { + if (first_p) + first_p = false; + else + return false; + } + + return true; +} + /* Return the successors of INSN in SUCCSP and their number in NP, honoring FLAGS. Empty blocks are skipped. */ void @@ -2640,7 +2994,7 @@ cfg_preds_1 (basic_block bb, insn_t **preds, int *n, int *size) /* ??? This code is not supposed to walk out of a region. */ gcc_assert (in_current_region_p (pred_bb)); - if (bb_note (pred_bb) == bb_end) + if (sel_bb_empty_p (pred_bb)) cfg_preds_1 (pred_bb, preds, n, size); else { @@ -2675,7 +3029,7 @@ num_preds_gt_1 (insn_t insn) { basic_block bb; - if (!bb_header_p (insn) || INSN_BB (insn) == 0) + if (!sel_bb_header_p (insn) || INSN_BB (insn) == 0) return false; bb = BLOCK_FOR_INSN (insn); @@ -2703,20 +3057,53 @@ num_preds_gt_1 (insn_t insn) gcc_assert (EDGE_PRED (bb, 0)->dest == bb); bb = EDGE_PRED (bb, 0)->src; - if (bb_note (bb) != BB_END (bb)) + if (!sel_bb_empty_p (bb)) break; } return false; } -/* True when BB consists of a single jump. */ -static bool -bb_jump_only_p (basic_block bb) +/* Returns true if INSN is not a downward continuation of the given path P in + the current stage. */ +bool +is_ineligible_successor (insn_t insn, ilist_t p) { - rtx jump = BB_END (bb); + insn_t prev_insn; + + /* Check if insn is not deleted. */ + if (PREV_INSN (insn) && NEXT_INSN (PREV_INSN (insn)) != insn) + gcc_unreachable (); + else if (NEXT_INSN (insn) && PREV_INSN (NEXT_INSN (insn)) != insn) + gcc_unreachable (); - return NEXT_INSN (bb_note (bb)) == jump && INSN_SIMPLEJUMP_P (jump); + /* If it's the first insn visited, then the successor is ok. */ + if (!p) + return false; + + prev_insn = ILIST_INSN (p); + + if (/* a backward edge. */ + INSN_SEQNO (insn) < INSN_SEQNO (prev_insn) + /* is already visited. */ + || (INSN_SEQNO (insn) == INSN_SEQNO (prev_insn) + && (ilist_is_in_p (p, insn) + /* We can reach another fence here and still seqno of insn + would be equal to seqno of prev_insn. This is possible + when prev_insn is a previously created bookkeeping copy. + In that case it'd get a seqno of insn. Thus, check here + whether insn is in current fence too. */ + || IN_CURRENT_FENCE_P (insn))) + /* Was already scheduled on this round. */ + || (INSN_SEQNO (insn) > INSN_SEQNO (prev_insn) + && IN_CURRENT_FENCE_P (insn)) + /* An insn from another fence could also be + scheduled earlier even if this insn is not in + a fence list right now. Check INSN_SCHED_CYCLE instead. */ + || (!pipelining_p && INSN_SCHED_TIMES (insn) > 0)) + return true; + else + return false; } /* Returns true when BB should be the end of an ebb. Adapted from the @@ -2772,46 +3159,20 @@ in_same_ebb_p (insn_t insn, insn_t succ) return false; } -/* Returns true if INSN is not a downward continuation of the given path P in - the current stage. */ -bool -is_ineligible_successor (insn_t insn, ilist_t p) +/* An implementation of create_basic_block hook, which additionally updates + per-bb data structures. */ +basic_block +sel_create_basic_block (void *headp, void *endp, basic_block after) { - insn_t prev_insn; - - /* Check if insn is not deleted. */ - if (PREV_INSN (insn) && NEXT_INSN (PREV_INSN (insn)) != insn) - gcc_unreachable (); - else if (NEXT_INSN (insn) && PREV_INSN (NEXT_INSN (insn)) != insn) - gcc_unreachable (); - - /* If it's the first insn visited, then the successor is ok. */ - if (!p) - return false; + basic_block new_bb; + + gcc_assert (flag_sel_sched_pipelining_outer_loops + || last_added_blocks == NULL); - prev_insn = ILIST_INSN (p); + new_bb = old_create_basic_block (headp, endp, after); + VEC_safe_push (basic_block, heap, last_added_blocks, new_bb); - if (/* a backward edge. */ - INSN_SEQNO (insn) < INSN_SEQNO (prev_insn) - /* is already visited. */ - || (INSN_SEQNO (insn) == INSN_SEQNO (prev_insn) - && (ilist_is_in_p (p, insn) - /* We can reach another fence here and still seqno of insn - would be equal to seqno of prev_insn. This is possible - when prev_insn is a previously created bookkeeping copy. - In that case it'd get a seqno of insn. Thus, check here - whether insn is in current fence too. */ - || IN_CURRENT_FENCE_P (insn))) - /* Was already scheduled on this round. */ - || (INSN_SEQNO (insn) > INSN_SEQNO (prev_insn) - && IN_CURRENT_FENCE_P (insn)) - /* An insn from another fence could also be - scheduled earlier even if this insn is not in - a fence list right now. Check INSN_SCHED_CYCLE instead. */ - || (!pipelining_p && INSN_SCHED_CYCLE (insn) > 0)) - return true; - else - return false; + return new_bb; } /* Recomputes the reverse topological order for the function and @@ -2877,13 +3238,13 @@ find_place_to_insert_bb (basic_block bb, int rgn) return (i + 1) - 1; } -/* Add (or remove depending on ADD_P) BB to (from) the current region +/* Add (or remove depending on ADD) BB to (from) the current region and update sched-rgn.c data. */ -void -sel_add_or_remove_bb_1 (basic_block bb, bool add_p) +static void +sel_add_or_remove_bb_1 (basic_block bb, int add) { int i, pos, bbi = -2, rgn; - int step = add_p ? 1 : 0; + int step = (add > 0) ? 1 : 0; rgn = CONTAINING_RGN (BB_TO_BLOCK (0)); @@ -2923,12 +3284,11 @@ sel_add_or_remove_bb_1 (basic_block bb, bool add_p) else { if (EDGE_COUNT (bb->succs) > 0) + /* We don't have preds outside the region. We should have + the only pred, because the multiple preds case comes from + the pipelining of outer loops, and that is handled above. + Just take the bbi of this single pred. */ { - /* We don't have preds outside the region. We should have the only - pred, because the multiple preds case comes from the pipelining of - outer loops, and that is handled above. Just take the bbi of - this single pred. */ - int pred_bbi; gcc_assert (EDGE_COUNT (bb->preds) == 1); @@ -2995,67 +3355,113 @@ sel_add_or_remove_bb_1 (basic_block bb, bool add_p) } } -/* Add (remove depending on ADD_P) BB to (from) the current region +/* Add (remove depending on ADD) BB to (from) the current region and update all data. If BB is NULL, add all blocks from last_added_blocks vector. */ void -sel_add_or_remove_bb (basic_block bb, bool add_p) +sel_add_or_remove_bb (basic_block bb, int add) { - bool b; - - if (add_p) + if (add > 0) { + /* Extend luids so that new notes will recieve zero luids. */ + sched_init_luids (NULL, NULL, NULL, NULL); + sched_init_bbs (last_added_blocks, NULL); + sel_init_bbs (last_added_blocks, NULL); + /* When bb is passed explicitly, the vector should contain the only element that equals to bb; otherwise, the vector should not be NULL. */ gcc_assert (last_added_blocks != NULL); - if (bb) + if (bb != NULL) { gcc_assert (VEC_length (basic_block, last_added_blocks) == 1 && VEC_index (basic_block, last_added_blocks, 0) == bb); + + /* Free the vector. */ VEC_free (basic_block, heap, last_added_blocks); } } else - gcc_assert (BB_NOTE_LIST (bb) == NULL_RTX); + { + gcc_assert (bb != NULL && BB_NOTE_LIST (bb) == NULL_RTX); + + if (glat_start[bb->index] != NULL) + FREE_REG_SET (glat_start[bb->index]); + if (glat_end[bb->index] != NULL) + FREE_REG_SET (glat_end[bb->index]); + } + + if (bb != NULL) + { + sel_add_or_remove_bb_1 (bb, add); - if (bb) - sel_add_or_remove_bb_1 (bb, add_p); + if (add < 0) + delete_basic_block (bb); + } else + /* BB is NULL - process LAST_ADDED_BLOCKS instead. */ { int i; basic_block temp_bb = NULL; - /* It is enough to call init once per vector. */ - gcc_assert (add_p); + gcc_assert (add > 0); for (i = 0; VEC_iterate (basic_block, last_added_blocks, i, bb); i++) { - sel_add_or_remove_bb_1 (bb, add_p); + sel_add_or_remove_bb_1 (bb, add); temp_bb = bb; } /* We need to fetch at least one bb so we know the region to update. */ + gcc_assert (temp_bb != NULL); bb = temp_bb; + VEC_free (basic_block, heap, last_added_blocks); } - if (!add_p) - delete_basic_block (bb); + rgn_setup_region (CONTAINING_RGN (bb->index)); +} + +/* A wrapper for create_basic_block_before, which also extends per-bb + data structures. Returns the newly created bb. */ +basic_block +sel_create_basic_block_before (basic_block before) +{ + basic_block prev_bb; + basic_block bb; + edge e; + + gcc_assert (in_current_region_p (before)); + + prev_bb = before->prev_bb; + + e = find_fallthru_edge (prev_bb); + gcc_assert (e != NULL); + + /* This code is taken from cfghooks.c: split_block (). */ + bb = create_basic_block (BB_HEAD (before), NULL_RTX, prev_bb); + bb->count = prev_bb->count; + bb->frequency = prev_bb->frequency; + bb->loop_depth = prev_bb->loop_depth; + make_single_succ_edge (bb, before, EDGE_FALLTHRU); + + redirect_edge_succ (e, bb); + + sel_add_or_remove_bb (bb, 1); - b = sched_rgn_local_init (CONTAINING_RGN (bb->index), true); - gcc_assert (!b); + return bb; } /* Remove an empty basic block EMPTY_BB. When MERGE_UP_P is true, we put EMPTY_BB's note lists into its predecessor instead of putting them into the successor. */ void -sel_remove_empty_bb (basic_block empty_bb, bool merge_up_p) +sel_remove_empty_bb (basic_block empty_bb, bool merge_up_p, + bool remove_from_cfg_p) { basic_block merge_bb; @@ -3095,8 +3501,9 @@ sel_remove_empty_bb (basic_block empty_bb, bool merge_up_p) && true)); /* If basic block has predecessors or successors, redirect them. */ - if (EDGE_COUNT (empty_bb->preds) > 0 - || EDGE_COUNT (empty_bb->succs) > 0) + if (remove_from_cfg_p + && (EDGE_COUNT (empty_bb->preds) > 0 + || EDGE_COUNT (empty_bb->succs) > 0)) { basic_block pred; basic_block succ; @@ -3133,7 +3540,6 @@ sel_remove_empty_bb (basic_block empty_bb, bool merge_up_p) else succ = NULL; - if (EDGE_COUNT (empty_bb->preds) > 0 && succ != NULL) redirect_edge_succ_nodup (EDGE_PRED (empty_bb, 0), succ); @@ -3147,39 +3553,48 @@ sel_remove_empty_bb (basic_block empty_bb, bool merge_up_p) } /* Finish removing. */ - sel_add_or_remove_bb (empty_bb, false); + sel_add_or_remove_bb (empty_bb, remove_from_cfg_p ? -1 : 0); } -/* Splits BB on two basic blocks, adding it to the region and extending - per-bb data structures. Returns the newly created bb. */ -basic_block -sel_split_block (basic_block bb, insn_t after) +/* Update the latch when we've splitted or merged it. + This should be checked for all outer loops, too. */ +static void +change_loops_latches (basic_block from, basic_block to) { - basic_block new_bb; - - insn_init.what = INSN_INIT_WHAT_INSN; - insn_init.todo = ((INSN_INIT_TODO_ALL & ~INSN_INIT_TODO_SEQNO) - | INSN_INIT_TODO_LV_SET); + gcc_assert (from != to); - new_bb = split_block (bb, after)->dest; - - /* Update the latch when we've splitted it. This should be checked - for all outer loops, too. */ if (flag_sel_sched_pipelining_outer_loops && current_loop_nest) { - struct loop * loop; + struct loop *loop; for (loop = current_loop_nest; loop; loop = loop->outer) - if (considered_for_pipelining_p (loop) && bb == loop->latch) + if (considered_for_pipelining_p (loop) && loop->latch == from) { gcc_assert (loop == current_loop_nest); - loop->latch = new_bb; + loop->latch = to; gcc_assert (loop_latch_edge (loop)); } } +} + +/* Splits BB on two basic blocks, adding it to the region and extending + per-bb data structures. Returns the newly created bb. */ +basic_block +sel_split_block (basic_block bb, insn_t after) +{ + basic_block new_bb; + + can_add_real_insns_p = false; + new_bb = split_block (bb, after)->dest; + can_add_real_insns_p = true; + + change_loops_latches (bb, new_bb); + + sel_add_or_remove_bb (new_bb, 1); + + gcc_assert (after != NULL || sel_bb_empty_p (bb)); - sel_add_or_remove_bb (new_bb, true); return new_bb; } @@ -3195,8 +3610,6 @@ sel_split_edge (edge e) && in_current_region_p (e->dest)); insn_init.what = INSN_INIT_WHAT_INSN; - insn_init.todo = INSN_INIT_TODO_ALL | INSN_INIT_TODO_LV_SET; - insn_init.how = INSN_INIT_HOW_DEFERRED; new_bb = split_edge (e); @@ -3206,8 +3619,8 @@ sel_split_edge (edge e) int i; basic_block bb; - /* Some of the basic blocks could not have been added to the loop. - Do this now, until this is fixed in force_fallthru. */ + /* Some of the basic blocks might not have been added to the loop. + Add them here, until this is fixed in force_fallthru. */ for (i = 0; VEC_iterate (basic_block, last_added_blocks, i, bb); i++) if (!bb->loop_father) @@ -3219,11 +3632,24 @@ sel_split_edge (edge e) /* Now the CFG has been updated, and we can init data for the newly created insns. */ - sel_insn_deferred_init (); + insn_init.todo = (INSN_INIT_TODO_LUID | INSN_INIT_TODO_SIMPLEJUMP); + sel_init_new_insns (); return new_bb; } +/* Merge basic block B into basic block A. */ +void +sel_merge_blocks (basic_block a, basic_block b) +{ + gcc_assert (can_merge_blocks_p (a, b)); + + sel_remove_empty_bb (b, true, false); + merge_blocks (a, b); + + change_loops_latches (b, a); +} + /* A wrapper for redirect_edge_and_branch_force, which also initializes data structures for possibly created bb and insns. Returns the newly added bb or NULL, when a bb was not needed. */ @@ -3232,27 +3658,23 @@ sel_redirect_edge_force (edge e, basic_block to) { basic_block jump_bb; - gcc_assert (bb_note (e->src) != BB_END (e->src)); + gcc_assert (!sel_bb_empty_p (e->src)); - insn_init.how = INSN_INIT_HOW_DEFERRED; jump_bb = redirect_edge_and_branch_force (e, to); if (jump_bb) - sel_add_or_remove_bb (jump_bb, true); - - /* Now the CFG has been updated, and we can init data for the newly - created insns. */ - sel_insn_deferred_init (); - - /* Only after sel_insn_deferred_init () we can check for - INSN_SIMPLEJUMP_P (). */ - gcc_assert (jump_bb == NULL || bb_jump_only_p (jump_bb)); + sel_add_or_remove_bb (jump_bb, 1); /* This function could not be used to spoil the loop structure by now, thus we don't care to update anything. But check it to be sure. */ if (flag_sel_sched_pipelining_outer_loops && current_loop_nest) gcc_assert (loop_latch_edge (current_loop_nest)); + /* Now the CFG has been updated, and we can init data for the newly + created insns. */ + insn_init.todo = (INSN_INIT_TODO_LUID | INSN_INIT_TODO_SIMPLEJUMP); + sel_init_new_insns (); + return jump_bb; } @@ -3278,44 +3700,89 @@ sel_redirect_edge_and_branch (edge e, basic_block to) gcc_assert (ee == e && last_added_blocks == NULL); + /* Now the CFG has been updated, and we can init data for the newly + created insns. */ + insn_init.todo = (INSN_INIT_TODO_LUID | INSN_INIT_TODO_SIMPLEJUMP); + sel_init_new_insns (); + return ee; } -/* A wrapper for create_basic_block_before, which also extends per-bb - data structures. Returns the newly created bb. */ -basic_block -sel_create_basic_block_before (basic_block before) + + +/* Emit an insn rtx based on PATTERN. */ +static rtx +create_insn_rtx_from_pattern_1 (rtx pattern) { - basic_block prev_bb; - basic_block bb; - edge e; + rtx insn_rtx; - gcc_assert (in_current_region_p (before)); + gcc_assert (!INSN_P (pattern)); - prev_bb = before->prev_bb; + start_sequence (); + insn_init.what = INSN_INIT_WHAT_INSN_RTX; + insn_rtx = emit_insn (pattern); + end_sequence (); - e = find_fallthru_edge (prev_bb); - gcc_assert (e != NULL); + sched_init_luids (NULL, NULL, NULL, NULL); + sel_extend_insn_rtx_data (); - /* This code is taken from cfghooks.c: split_block (). */ - bb = create_basic_block (BB_HEAD (before), NULL_RTX, prev_bb); - bb->count = prev_bb->count; - bb->frequency = prev_bb->frequency; - bb->loop_depth = prev_bb->loop_depth; - make_single_succ_edge (bb, before, EDGE_FALLTHRU); + return insn_rtx; +} - redirect_edge_succ (e, bb); +/* Emit an insn rtx based on PATTERN and ICE if the result is not a valid + insn. */ +rtx +create_insn_rtx_from_pattern (rtx pattern) +{ + rtx insn_rtx = create_insn_rtx_from_pattern_1 (pattern); - sel_add_or_remove_bb (bb, true); + if (!insn_rtx_valid (insn_rtx)) + gcc_unreachable (); - return bb; + return insn_rtx; } - -/* Helpers for global init. */ -/* Data structure to describe interaction with the generic scheduler utils. */ -static struct common_sched_info_def sel_common_sched_info; +/* Create a new vinsn for INSN_RTX. */ +vinsn_t +create_vinsn_from_insn_rtx (rtx insn_rtx) +{ + gcc_assert (INSN_P (insn_rtx) && !INSN_IN_STREAM_P (insn_rtx)); + + return vinsn_create (insn_rtx, false); +} + +/* Create a copy of INSN_RTX. */ +rtx +create_copy_of_insn_rtx (rtx insn_rtx) +{ + bool orig_is_valid_p; + rtx res; + + gcc_assert (INSN_P (insn_rtx)); + + orig_is_valid_p = insn_rtx_valid (insn_rtx); + + res = create_insn_rtx_from_pattern_1 (copy_rtx (PATTERN (insn_rtx))); + + if (insn_rtx_valid (res)) + gcc_assert (orig_is_valid_p); + else + gcc_assert (!orig_is_valid_p); + return res; +} + +/* Change vinsn field of EXPR to hold NEW_VINSN. */ +void +change_vinsn_in_expr (expr_t expr, vinsn_t new_vinsn) +{ + vinsn_detach (EXPR_VINSN (expr)); + + EXPR_VINSN (expr) = new_vinsn; + vinsn_attach (new_vinsn); +} + +/* Helpers for global init. */ /* This structure is used to be able to call existing bundling mechanism and calculate insn priorities. */ static struct haifa_sched_info sched_sel_haifa_sched_info = @@ -3325,7 +3792,7 @@ static struct haifa_sched_info sched_sel_haifa_sched_info = NULL, /* schedule_more_p */ NULL, /* new_ready */ NULL, /* rgn_rank */ - sel_print_insn, /* rgn_print_insn */ + NULL, /* rgn_print_insn */ contributes_to_priority, NULL, NULL, @@ -3341,52 +3808,74 @@ static struct haifa_sched_info sched_sel_haifa_sched_info = void setup_nop_and_exit_insns (void) { - if (!nop_pattern) + if (nop_pattern == NULL_RTX) nop_pattern = gen_nop (); if (exit_insn == NULL_RTX) { start_sequence (); + insn_init.what = INSN_INIT_WHAT_INSN_RTX; emit_insn (nop_pattern); exit_insn = get_insns (); end_sequence (); } + set_block_for_insn (exit_insn, EXIT_BLOCK_PTR); } +/* Free special insns used in the scheduler. */ void -free_exit_insn_data (void) +free_nop_and_exit_insns (void) { - gcc_assert (LV_SET_VALID_P (exit_insn)); - return_regset_to_pool (LV_SET (exit_insn)); - exit_insn = NULL_RTX; + nop_pattern = NULL_RTX; +} +/* Setup a special vinsn used in new insns initialization. */ +void +setup_empty_vinsn (void) +{ + empty_vinsn = vinsn_create (exit_insn, false); + vinsn_attach (empty_vinsn); } -/* Setup pointers to global sched info structures. */ +/* Free a special vinsn used in new insns initialization. */ +void +free_empty_vinsn (void) +{ + gcc_assert (VINSN_COUNT (empty_vinsn) == 1); + vinsn_detach (empty_vinsn); + empty_vinsn = NULL; +} + +/* Data structure to describe interaction with the generic scheduler utils. */ +static struct common_sched_info_def sel_common_sched_info; + +/* Setup common_sched_info. */ void -setup_sched_and_deps_infos (void) +sel_setup_common_sched_info (void) { - memcpy (&sel_common_sched_info, &rgn_common_sched_info, + rgn_setup_common_sched_info (); + + memcpy (&sel_common_sched_info, common_sched_info, sizeof (sel_common_sched_info)); - sel_common_sched_info.sched_pass_id = SCHED_SEL_PASS; + sel_common_sched_info.fix_recovery_cfg = NULL; sel_common_sched_info.add_block = NULL; sel_common_sched_info.estimate_number_of_insns - = sched_sel_estimate_number_of_insns; - sel_common_sched_info.which_luid = sched_sel_which_luid; - sel_common_sched_info.remove_notes = sel_remove_notes; - sel_common_sched_info.bb_extend = sel_extend_bb_info; - sel_common_sched_info.bb_finish = sel_finish_bb_info; - sel_common_sched_info.insn_extend = sel_extend_insn_info; - sel_common_sched_info.insn_init = sel_init_insn_info; - sel_common_sched_info.insn_finish = sel_finish_insn_info; + = sel_estimate_number_of_insns; + sel_common_sched_info.luid_for_non_insn = sel_luid_for_non_insn; + sel_common_sched_info.detach_life_info = 1; + sel_common_sched_info.sched_pass_id = SCHED_SEL_PASS; common_sched_info = &sel_common_sched_info; - current_sched_info = &sched_sel_haifa_sched_info; +} - sched_deps_info = &sel_sched_deps_info; +/* Setup pointers to global sched info structures. */ +void +sel_setup_sched_infos (void) +{ + current_sched_info = &sched_sel_haifa_sched_info; } /* Adds basic block BB to region RGN at the position *BB_ORD_INDEX, @@ -3397,6 +3886,9 @@ sel_add_block_to_region (basic_block bb, int *bb_ord_index, int rgn) RGN_NR_BLOCKS (rgn) += 1; RGN_DONT_CALC_DEPS (rgn) = 0; RGN_HAS_REAL_EBB (rgn) = 0; + RGN_HAS_RENAMING_P (nr_regions) = 0; + RGN_WAS_PIPELINED_P (nr_regions) = 0; + RGN_NEEDS_GLOBAL_LIVE_UPDATE (nr_regions) = 0; CONTAINING_RGN (bb->index) = rgn; BLOCK_TO_BB (bb->index) = *bb_ord_index; rgn_bb_table[RGN_BLOCKS (rgn) + *bb_ord_index] = bb->index; @@ -3406,6 +3898,7 @@ sel_add_block_to_region (basic_block bb, int *bb_ord_index, int rgn) RGN_BLOCKS (rgn + 1) = RGN_BLOCKS (rgn) + RGN_NR_BLOCKS (rgn); } +/* Functions to support pipelining of outer loops. */ /* Creates a new empty region and returns it's number. */ static int @@ -3690,6 +4183,23 @@ make_regions_from_the_rest (void) extend_rgns (degree, &cur_rgn_blocks, bbs_in_loop_rgns, loop_hdr); + /* Any block that did not end up in a region is placed into a region + by itself. */ + FOR_EACH_BB (bb) + if (degree[bb->index] >= 0) + { + rgn_bb_table[cur_rgn_blocks] = bb->index; + RGN_NR_BLOCKS (nr_regions) = 1; + RGN_BLOCKS (nr_regions) = cur_rgn_blocks++; + RGN_DONT_CALC_DEPS (nr_regions) = 0; + RGN_HAS_REAL_EBB (nr_regions) = 0; + RGN_HAS_RENAMING_P (nr_regions) = 0; + RGN_WAS_PIPELINED_P (nr_regions) = 0; + RGN_NEEDS_GLOBAL_LIVE_UPDATE (nr_regions) = 0; + CONTAINING_RGN (bb->index) = nr_regions++; + BLOCK_TO_BB (bb->index) = 0; + } + free (degree); free (loop_hdr); } @@ -3755,11 +4265,81 @@ sel_add_loop_preheader (void) sel_add_or_remove_bb_1 (bb, true); /* Set variables for the current region. */ - sched_rgn_local_preinit (rgn); + rgn_setup_region (rgn); } VEC_free (basic_block, heap, preheader_blocks); MARK_LOOP_FOR_PIPELINING (current_loop_nest); } +/* While pipelining outer loops, returns TRUE if BB is a loop preheader. */ +bool +sel_is_loop_preheader_p (basic_block bb) +{ + /* A preheader may even have the loop depth equal to the depth of + the current loop, when it came from it. Use topological sorting + to get the right information. */ + if (flag_sel_sched_pipelining_outer_loops + && current_loop_nest) + { + struct loop *outer; + /* BB is placed before the header, so, it is a preheader block. */ + if (BLOCK_TO_BB (bb->index) + < BLOCK_TO_BB (current_loop_nest->header->index)) + return true; + + /* Support the situation when the latch block of outer loop + could be from here. */ + for (outer = current_loop_nest->outer; outer; outer = outer->outer) + if (considered_for_pipelining_p (outer) && outer->latch == bb) + gcc_unreachable (); + } + return false; +} + +/* Removes the loop preheader from the current region and saves it in + PREHEADER_BLOCKS of the father loop, so they will be added later to + region that represents an outer loop. + This function is only used with -fsel-sched-pipelining-outer-loops. */ +static void +sel_remove_loop_preheader (void) +{ + int i, old_len; + int cur_rgn = CONTAINING_RGN (BB_TO_BLOCK (0)); + basic_block bb; + VEC(basic_block, heap) *preheader_blocks + = LOOP_PREHEADER_BLOCKS (current_loop_nest->outer); + + gcc_assert (flag_sel_sched_pipelining_outer_loops && current_loop_nest); + old_len = VEC_length (basic_block, preheader_blocks); + + /* Add blocks that aren't within the current loop to PREHEADER_BLOCKS. */ + for (i = 0; i < RGN_NR_BLOCKS (cur_rgn); i++) + { + bb = BASIC_BLOCK (BB_TO_BLOCK (i)); + + /* If the basic block belongs to region, but doesn't belong to + corresponding loop, then it should be a preheader. */ + if (sel_is_loop_preheader_p (bb)) + VEC_safe_push (basic_block, heap, preheader_blocks, bb); + } + + /* Remove these blocks only after iterating over the whole region. */ + for (i = VEC_length (basic_block, preheader_blocks) - 1; + i >= old_len; + i--) + { + bb = VEC_index (basic_block, preheader_blocks, i); + + sel_add_or_remove_bb (bb, 0); + } + + if (!considered_for_pipelining_p (current_loop_nest->outer)) + /* Immediately create new region from preheader. */ + make_region_from_loop_preheader (&preheader_blocks); + else + /* Store preheader within the father's loop structure. */ + SET_LOOP_PREHEADER_BLOCKS (current_loop_nest->outer, preheader_blocks); +} + #endif diff --git a/gcc/sel-sched-ir.h b/gcc/sel-sched-ir.h index a8c1cd92a67..0a6b6928fdf 100644 --- a/gcc/sel-sched-ir.h +++ b/gcc/sel-sched-ir.h @@ -78,10 +78,9 @@ typedef _xlist_t ilist_t; /* Expression macros -- to be removed. */ #define expr_equal_p(A, B) (rtx_equal_p (A, B)) -#define expr_copy(EXPR) (copy_rtx (EXPR)) /* Right hand side information. */ -struct _rhs +struct _expr { /* Insn description. */ vinsn_t vinsn; @@ -95,34 +94,60 @@ struct _rhs /* A priority of this expression. */ int priority; + + /* Number of times the insn was scheduled. */ + int sched_times; + + /* Instruction should be of SPEC_DONE_DS type in order to be moved to this + point. */ + ds_t spec_done_ds; + + /* SPEC_TO_CHECK_DS hold speculation types that should be checked + (used only during move_op ()). */ + ds_t spec_to_check_ds; }; -/* Right hand side. */ -typedef struct _rhs *rhs_t; +typedef struct _expr expr_def; +typedef expr_def *expr_t; + +/* Obsolete. */ +typedef expr_t rhs_t; + +#define EXPR_VINSN(EXPR) ((EXPR)->vinsn) +#define EXPR_INSN_RTX(EXPR) (VINSN_INSN_RTX (EXPR_VINSN (EXPR))) +#define EXPR_PATTERN(EXPR) (VINSN_PATTERN (EXPR_VINSN (EXPR))) +#define EXPR_LHS(EXPR) (VINSN_LHS (EXPR_VINSN (EXPR))) +#define EXPR_RHS(EXPR) (VINSN_RHS (EXPR_VINSN (EXPR))) +#define EXPR_TYPE(EXPR) (VINSN_TYPE (EXPR_VINSN (EXPR))) +#define EXPR_SEPARABLE_P(EXPR) (VINSN_SEPARABLE_P (EXPR_VINSN (EXPR))) + +#define EXPR_SPEC(EXPR) ((EXPR)->spec) +#define EXPR_PRIORITY(EXPR) ((EXPR)->priority) +#define EXPR_SCHED_TIMES(EXPR) ((EXPR)->sched_times) +#define EXPR_SPEC_DONE_DS(EXPR) ((EXPR)->spec_done_ds) +#define EXPR_SPEC_TO_CHECK_DS(EXPR) ((EXPR)->spec_to_check_ds) + +/* Obsolete. */ #define RHS_VINSN(RHS) ((RHS)->vinsn) -#define RHS_SPEC(RHS) ((RHS)->spec) #define RHS_INSN(RHS) (VINSN_INSN (RHS_VINSN (RHS))) -#define RHS_PAT(RHS) (PATTERN (RHS_INSN (RHS))) +#define RHS_PATTERN(RHS) (VINSN_PATTERN (RHS_VINSN (RHS))) +#define RHS_SPEC(RHS) ((RHS)->spec) #define RHS_PRIORITY(RHS) ((RHS)->priority) -#define RHS_DEST(RHS) (VINSN_LHS (RHS_VINSN (RHS))) - -/* FIXME: rename it!!! */ -#define RHS_SCHEDULE_AS_RHS(RHS) (VINSN_SEPARABLE (RHS_VINSN (RHS))) -#define RHS_HAS_RHS(RHS) (VINSN_HAS_RHS (RHS_VINSN (RHS))) - +#define RHS_SCHED_TIMES(RHS) ((RHS)->sched_times) /* Insn definition for list of original insns in find_used_regs. */ struct _def { insn_t orig_insn; bool crosses_call; + bool needs_spec_check_p; }; typedef struct _def *def_t; /* Availability sets arae sets of expressions we're scheduling. */ typedef _list_t av_set_t; -#define _AV_SET_RHS(L) (&(L)->u.rhs) +#define _AV_SET_EXPR(L) (&(L)->u.expr) #define _AV_SET_NEXT(L) (_LIST_NEXT (L)) @@ -240,7 +265,7 @@ struct _list_node { rtx x; struct _bnd bnd; - struct _rhs rhs; + expr_def expr; struct _fence fence; struct _def def; void *data; @@ -392,7 +417,7 @@ _list_iter_cond_rhs (av_set_t av, rhs_t *rhsp) { if (av) { - *rhsp = _AV_SET_RHS (av); + *rhsp = _AV_SET_EXPR (av); return true; } @@ -426,8 +451,16 @@ _list_iter_cond_def (def_list_t def_list, def_t *def) /* InstructionData. Contains information about insn pattern. */ struct idata_def { - /* Type of the insn. Can be INSN, CALL_INSN, JUMP_INSN, ... something - else and SET. If it is SET, then insn can be cloned. */ + /* Type of the insn. + o CALL_INSN - Call insn + o JUMP_INSN - Jump insn + o INSN - INSN that cannot be cloned + o USE - INSN that can be cloned + o SET - INSN that can be cloned and separable into lhs and rhs + o PC - simplejump. Insns that simply redirect control flow should not + have any dependencies. Sched-deps.c, though, might consider them as + producers or consumers of certain registers. To avoid that we handle + dependency for simple jumps ourselves. */ int type; /* If insn is a SET, this is its left hand side. */ @@ -448,6 +481,9 @@ struct idata_def regset reg_sets; regset reg_uses; + + /* I hope that our renaming infrastructure handles this. */ + /* bool has_internal_dep; */ }; #define IDATA_TYPE(ID) ((ID)->type) @@ -455,7 +491,6 @@ struct idata_def #define IDATA_RHS(ID) ((ID)->rhs) #define IDATA_REG_SETS(ID) ((ID)->reg_sets) #define IDATA_REG_USES(ID) ((ID)->reg_uses) -#define IDATA_SIMPLEJUMP_P(ID) ((ID)->simplejump_p) /* Type to represent all needed info to emit an insn. This is a virtual equivalent of the insn. @@ -470,48 +505,55 @@ struct idata_def struct vinsn_def { /* Associated insn. */ - insn_t insn; + rtx insn_rtx; /* Its description. */ idata_t id; - /* Time of schedule. It is greater than zero if insn was scheduled. */ - int sched_cycle; - - /* Number of times the insn was scheduled. */ - int sched_times; - /* Smart pointer counter. */ int count; - /* Whether we will separate insn and schedule it as RHS. */ - bool separable; + /* Cached cost of the vinsn. To access it please use vinsn_cost (). */ + int cost; + + /* Mark insns that may trap so we don't move them through jumps. */ + bool may_trap_p; }; +#define VINSN_INSN_RTX(VI) ((VI)->insn_rtx) +#define VINSN_PATTERN(VI) (PATTERN (VINSN_INSN_RTX (VI))) +/* Obsolete. */ +#define VINSN_INSN(VI) (VINSN_INSN_RTX (VI)) #define VINSN_ID(VI) ((VI)->id) #define VINSN_TYPE(VI) (IDATA_TYPE (VINSN_ID (VI))) -#define VINSN_INSN(VI) ((VI)->insn) +#define VINSN_SEPARABLE_P(VI) (VINSN_TYPE (VI) == SET) +#define VINSN_CLONABLE_P(VI) (VINSN_SEPARABLE_P (VI) || VINSN_TYPE (VI) == USE) +#define VINSN_UNIQUE_P(VI) (!VINSN_CLONABLE_P (VI)) #define VINSN_LHS(VI) (IDATA_LHS (VINSN_ID (VI))) #define VINSN_RHS(VI) (IDATA_RHS (VINSN_ID (VI))) #define VINSN_REG_SETS(VI) (IDATA_REG_SETS (VINSN_ID (VI))) #define VINSN_REG_USES(VI) (IDATA_REG_USES (VINSN_ID (VI))) -#define VINSN_SCHED_CYCLE(VI) ((VI)->sched_cycle) -#define VINSN_SCHED_TIMES(VI) ((VI)->sched_times) + #define VINSN_COUNT(VI) ((VI)->count) -#define VINSN_HAS_RHS(VI) (!!VINSN_RHS (VI)) -#define VINSN_SEPARABLE(VI) ((VI)->separable) -#define VINSN_PATTERN(VI) (PATTERN (VINSN_INSN (VI))) -#define VINSN_UNIQUE_P(VI) (VINSN_TYPE (VI) != SET) - +#define VINSN_MAY_TRAP_P(VI) ((VI)->may_trap_p) -/* Indexed by INSN_UID, the collection of all data associated with - a single instruction that is in the stream. - Though this data is indexed by INSN_UID, it should be indexed by - INSN_LUID. */ -struct sel_insn_data + + +/* Indexed by INSN_LUID, the collection of all data associated with + a single instruction that is in the stream. */ +struct _sel_insn_data { + /* Insn is an ASM. */ + bool asm_p; + + /* This field is initialized at the beginning of scheduling and is used + to handle sched group instructions. If it is non-null, then it points + to the instruction, which should be forced to schedule next. Such + instructions are unique. */ + insn_t sched_next; + /* Every insn in the stream has an associated vinsn. This is used to reduce memory consumption and give use the fact that many insns (all of them at the moment) don't change through the scheduler. @@ -519,81 +561,105 @@ struct sel_insn_data Much of the information, that reflect information about the insn itself (e.g. not about where it stands in CFG) is contained in the VINSN_ID field of this home VI. */ - vinsn_t vi; + expr_def _expr; - int seqno; int av_level; - int ws_level; av_set_t av; - /* This field is initialized at the beginning of scheduling and is used - to handle sched group instructions. If it is non-null, then it points - to the instruction, which should be forced to schedule next. Such - instructions are unique (i.e. can't be dublicated) as INSN_ASM_P is - forced for them. */ - insn_t sched_next; - - /* Insns that simply redirect control flow should not have any dependencies. - sched-deps.c , though, might consider them as producers or consumers of - certain registers. To avoid that we handle dependency for simple jumps - ourselves. */ - BOOL_BITFIELD simplejump_p : 1; - - /* ??? This should be called unique_p as it might be set not for asms - only. It simply means that insn can't be cloned. Such insns always - have only one vinsn associated - INSN_VI field above. */ - BOOL_BITFIELD asm_p : 1; + int seqno; - /* True when an insn is scheduled after we've determined that a stall is required. + /* True when an insn is scheduled after we've determined that a stall is + required. This is used when emulating the Haifa scheduler for bundling. */ BOOL_BITFIELD after_stall_p : 1; - /* Cycle at which insn was scheduled. This is used for bundling. */ - int final_sched_cycle; + /* Cycle at which insn was scheduled. It is greater than zero if insn was + scheduled. + This is used for bundling. */ + int sched_cycle; }; -extern struct sel_insn_data *s_i_d; +typedef struct _sel_insn_data sel_insn_data_def; +typedef sel_insn_data_def *sel_insn_data_t; + +DEF_VEC_O (sel_insn_data_def); +DEF_VEC_ALLOC_O (sel_insn_data_def, heap); +extern VEC (sel_insn_data_def, heap) *s_i_d; + +/* Accessor macros for s_s_i_d. */ +#define SID(INSN) (VEC_index (sel_insn_data_def, s_i_d, INSN_LUID (INSN))) + +#define INSN_ASM_P(INSN) (SID (INSN)->asm_p) +#define INSN_SCHED_NEXT(INSN) (SID (INSN)->sched_next) + +#define INSN_EXPR(INSN) (&SID (INSN)->_expr) +#define INSN_VINSN(INSN) (RHS_VINSN (INSN_EXPR (INSN))) +#define INSN_TYPE(INSN) (VINSN_TYPE (INSN_VINSN (INSN))) +#define INSN_SIMPLEJUMP_P(INSN) (INSN_TYPE (INSN) == PC) +#define INSN_LHS(INSN) (VINSN_LHS (INSN_VINSN (INSN))) +#define INSN_RHS(INSN) (VINSN_RHS (INSN_VINSN (INSN))) +#define INSN_REG_SETS(INSN) (VINSN_REG_SETS (INSN_VINSN (INSN))) +#define INSN_REG_USES(INSN) (VINSN_REG_USES (INSN_VINSN (INSN))) +#define INSN_SCHED_TIMES(INSN) (EXPR_SCHED_TIMES (INSN_EXPR (INSN))) +/* Obsolete. */ +#define INSN_VI(INSN) (INSN_VINSN (INSN)) + +#define INSN_AV_LEVEL(INSN) (SID (INSN)->av_level) +#define INSN_AV_VALID_P(INSN) (INSN_AV_LEVEL (INSN) == global_level) +/* Obsolete. */ +#define AV_LEVEL(INSN) (INSN_AV_LEVEL (INSN)) + +#define INSN_AV(INSN) (SID (INSN)->av) +/* Obsolete. */ +#define AV_SET(INSN) (INSN_AV (INSN)) + +#define INSN_SEQNO(INSN) (SID (INSN)->seqno) +#define INSN_AFTER_STALL_P(INSN) (SID (INSN)->after_stall_p) +#define INSN_SCHED_CYCLE(INSN) (SID (INSN)->sched_cycle) + /* A global level shows whether an insn is valid or not. */ extern int global_level; /* A list of fences currently in the works. */ extern flist_t fences; -/* Accessor macros for s_i_d. */ -#define SID(INSN) (s_i_d[INSN_UID (INSN)]) - -#define INSN_SEQNO(INSN) (SID (INSN).seqno) -#define AV_LEVEL(INSN) (SID (INSN).av_level) -#define AV_SET(INSN) (SID (INSN).av) -#define INSN_ASM_P(INSN) (SID (INSN).asm_p) - -#define INSN_VI(INSN) (SID (INSN).vi) -#define INSN_ID(INSN) (VINSN_ID (INSN_VI (INSN))) -#define INSN_TYPE(INSN) (IDATA_TYPE (INSN_ID (INSN))) -#define INSN_LHS(INSN) (IDATA_LHS (INSN_ID (INSN))) -#define INSN_RHS(INSN) (IDATA_RHS (INSN_ID (INSN))) -#define INSN_REG_SETS(INSN) (IDATA_REG_SETS (INSN_ID (INSN))) -#define INSN_REG_USES(INSN) (IDATA_REG_USES (INSN_ID (INSN))) +/* NB: All the fields, gathered in the s_s_i_d are initialized only for + insns that are in the current region and, therefore, can be indexed by + LUID. Except for these. . */ +struct _sel_insn_rtx_data +{ + /* For each bb header this field contains a set of live registers. + For all other insns this field has a NULL. + We also need to know LV sets for the instructions, that are immediatly + after the border of the region. */ + regset lv; + + /* Vinsn corresponding to this insn. + We need this field to be accessible for every instruction - not only + for those that have luids - because when choosing an instruction from + ready list there is no other fast way to get corresponding vinsn which + holds all the data for any given insn_rtx. */ + vinsn_t get_vinsn_by_insn; +}; -#define INSN_SCHED_CYCLE(INSN) (VINSN_SCHED_CYCLE (INSN_VI (INSN))) -#define INSN_FINAL_SCHED_CYCLE(INSN) (SID (INSN).final_sched_cycle) +typedef struct _sel_insn_rtx_data sel_insn_rtx_data_def; +typedef sel_insn_rtx_data_def *sel_insn_rtx_data_t; -#define INSN_SCHED_NEXT(INSN) (SID (INSN).sched_next) -#define INSN_SIMPLEJUMP_P(INSN) (SID (INSN).simplejump_p) -#define INSN_AFTER_STALL_P(INSN) (SID (INSN).after_stall_p) +DEF_VEC_O (sel_insn_rtx_data_def); +DEF_VEC_ALLOC_O (sel_insn_rtx_data_def, heap); -#define INSN_AV_VALID_P(INSN) (AV_LEVEL (INSN) == global_level) -#define INSN_UNIQUE_P(INSN) (INSN_TYPE (INSN) != SET) +extern VEC (sel_insn_rtx_data_def, heap) *s_i_r_d; -/* Points to liveness sets. */ -extern regset * lvs; +#define SIRD(INSN) \ +(VEC_index (sel_insn_rtx_data_def, s_i_r_d, INSN_UID (INSN))) /* Access macro. */ -#define LV_SET(INSN) (lvs[INSN_UID (INSN)]) +#define LV_SET(INSN) (SIRD (INSN)->lv) /* !!! Replace all occurencies with (LV_SET (INSN) != NULL). */ -#define LV_SET_VALID_P(INSN) (lvs[INSN_UID (INSN)] != NULL) +#define LV_SET_VALID_P(INSN) (LV_SET (INSN) != NULL) +#define GET_VINSN_BY_INSN(INSN) (SIRD (INSN)->get_vinsn_by_insn) -/* Size of the per insn data structures. */ -extern int sel_max_uid; +extern void sel_extend_insn_rtx_data (void); +extern void sel_finish_insn_rtx_data (void); /* A NOP pattern used as a placeholder for real insns. */ extern rtx nop_pattern; @@ -635,45 +701,19 @@ extern basic_block (*old_create_basic_block) (void *, void *, basic_block); /* Types used when initializing insn data. */ -/* !!! Think about this. It would be better to have separate sets of hooks - to initialize insns and vinsns. */ -enum insn_init_what { INSN_INIT_WHAT_INSN, INSN_INIT_WHAT_VINSN }; +enum insn_init_what { INSN_INIT_WHAT_INSN, INSN_INIT_WHAT_INSN_RTX }; -/* !!! Think about this. It would be better to have separate sets of hooks - to initialize (v)insns now or later. */ -enum insn_init_how { INSN_INIT_HOW_NOW, INSN_INIT_HOW_DEFERRED }; - -/* Do not initialize anything. */ -#define INSN_INIT_TODO_NOTHING (0) /* Provide a separate luid for the insn. */ #define INSN_INIT_TODO_LUID (1) -/* Initialize LHS. */ -#define INSN_INIT_TODO_LHS (2) -/* Initialize RHS. */ -#define INSN_INIT_TODO_RHS (4) -/* Initialize INSN_VI () field in s_i_d. */ -#define INSN_INIT_TODO_VINSN (8) -/* Initialize LV_SET for insn. */ -#define INSN_INIT_TODO_LV_SET (16) -/* Initialize all Insn Data. */ -#define INSN_INIT_TODO_INIT_ID (INSN_INIT_TODO_LHS | INSN_INIT_TODO_RHS \ - | INSN_INIT_TODO_VINSN) -/* Initialize fundamental instruction flags like SCHED_GROUP. This is being - run at the scheduler init. */ -#define INSN_INIT_TODO_GLOBAL (32) -/* Prepare for dependence analysis. */ -#define INSN_INIT_TODO_PREPARE_DEP (64) -/* Run dependence analysis. */ -#define INSN_INIT_TODO_HAS_DEP (128) -/* Provide a seqno for the insn. */ -#define INSN_INIT_TODO_SEQNO (256) -/* Init everything in s_i_d. */ -#define INSN_INIT_TODO_SID (512 | INSN_INIT_TODO_INIT_ID \ - | INSN_INIT_TODO_SEQNO) -/* Init luid and s_i_d. */ -#define INSN_INIT_TODO_ALL (INSN_INIT_TODO_LUID | INSN_INIT_TODO_SID) + +/* Initialize s_s_i_d. */ +#define INSN_INIT_TODO_SSID (2) + +/* Initialize LV_SET and SEQNO for simplejump. */ +#define INSN_INIT_TODO_SIMPLEJUMP (4) + /* Move LV_SET to the insn if it is being added to the bb header. */ -#define INSN_INIT_TODO_MOVE_LV_SET_IF_BB_HEADER (1024) +#define INSN_INIT_TODO_MOVE_LV_SET_IF_BB_HEADER (8) /* A container to hold information about insn initialization. */ @@ -682,75 +722,43 @@ struct _insn_init /* What is being initialized. */ enum insn_init_what what; - /* How this is being initialized. */ - enum insn_init_how how; - /* This is a set of INSN_INIT_TODO_* flags. */ int todo; - - /* Insn Data of the insn being initialized. - !!! Move it somewhere else. */ - idata_t id; - - /* Data for global dependency analysis (to initialize CANT_MOVE and - SCHED_GROUP_P). */ - struct - { - /* Current insn. */ - insn_t insn; - - /* Previous insn. */ - insn_t prev_insn; - - /* Deps context. */ - struct deps deps; - } global; - - /* Data to support deferred instruction initialization. */ - struct insn_init_how_deferred_def - { - /* Array of pending insns. */ - insn_t *insns; - - /* Number of elements in the above array. */ - int n; - - /* Size of the above array. */ - int size; - } how_deferred; }; extern struct _insn_init insn_init; /* A variable to track which part of rtx we are scanning in sched-deps.c: sched_analyze_insn (). */ -enum deps_where_t +enum _deps_where { DEPS_IN_INSN, DEPS_IN_LHS, DEPS_IN_RHS, DEPS_IN_NOWHERE }; - -/* Is SEL_DEPS_HAS_DEP_P[DEPS_IN_X] is true, then X has a dependence. - X is from { INSN, LHS, RHS }. */ -extern bool sel_deps_has_dep_p[]; +typedef enum _deps_where deps_where_t; /* Per basic block data. */ -struct sel_bb_info_def +struct _sel_bb_info { /* This insn stream is constructed in such a way that it should be traversed by PREV_INSN field - (*not* NEXT_INSN). */ rtx note_list; }; -typedef struct sel_bb_info_def *sel_bb_info_t; + +typedef struct _sel_bb_info sel_bb_info_def; +typedef sel_bb_info_def *sel_bb_info_t; + +DEF_VEC_O (sel_bb_info_def); +DEF_VEC_ALLOC_O (sel_bb_info_def, heap); /* Per basic block data. This array is indexed by basic block index. */ -extern sel_bb_info_t sel_bb_info; +extern VEC (sel_bb_info_def, heap) *sel_bb_info; /* Get data for BB. */ -#define SEL_BB_INFO(BB) (&sel_bb_info[(BB)->index]) +#define SEL_BB_INFO(BB) (VEC_index (sel_bb_info_def, sel_bb_info, (BB)->index)) /* Get BB's note_list. A note_list is a list of various notes that was scattered across BB @@ -773,10 +781,6 @@ extern bool enable_moveup_set_path_p; extern bool enable_schedule_as_rhs_p; extern bool pipelining_p; extern bool bookkeeping_p; -/* !!! Possibly remove: UNIQUE_RHSES are always used and UNIQUE_VINSNS are - never used. - Flags to pass to av_set_copy (). */ -enum { UNIQUE_RHSES = 1, UNIQUE_VINSNS }; /* Functions that are used in sel-sched.c. */ @@ -791,7 +795,7 @@ extern void flist_add (flist_t *, insn_t, state_t, deps_t, void *, extern fence_t flist_lookup (flist_t, insn_t); extern void flist_clear (flist_t *); -extern void def_list_add (def_list_t *, insn_t, bool); +extern void def_list_add (def_list_t *, insn_t, bool, bool); /* Target context functions. */ extern tc_t create_target_context (bool); @@ -816,79 +820,103 @@ extern void free_regset_pool (void); extern insn_t get_nop_from_pool (insn_t); extern void return_nop_to_pool (insn_t); +extern void free_nop_pool (void); /* Vinsns functions. */ extern bool vinsn_separable_p (vinsn_t); -extern void vinsn_attach (vinsn_t); -extern void vinsn_detach (vinsn_t); extern bool vinsn_cond_branch_p (vinsn_t); extern void recompute_vinsn_lhs_rhs (vinsn_t); +extern int sel_vinsn_cost (vinsn_t); +extern insn_t sel_gen_insn_from_rtx_after (rtx, expr_t, int, insn_t); +extern insn_t sel_gen_insn_from_expr_after (expr_t, int, insn_t); /* RHS functions. */ -extern bool rhs_equal_p (rhs_t, rhs_t); -extern void rhs_init (rhs_t, vinsn_t, int, int); -extern void rhs_copy (rhs_t, rhs_t); -extern void rhs_clear (rhs_t); +extern bool vinsns_correlate_as_rhses_p (vinsn_t, vinsn_t); +extern void copy_expr (expr_t, expr_t); +extern void merge_expr_data (expr_t, expr_t); +extern void merge_expr (expr_t, expr_t); +extern void clear_expr (expr_t); /* Av set functions. */ +extern void av_set_add (av_set_t *, rhs_t); extern void av_set_iter_remove (av_set_iterator *); -extern rhs_t av_set_lookup_rhs (av_set_t, rhs_t); -extern rhs_t av_set_lookup_other_equiv_rhs (av_set_t, rhs_t); -extern bool av_set_remove_rhs_with_insn (av_set_t *, insn_t); -extern bool av_set_is_in_p (av_set_t, rhs_t); -extern rhs_t av_set_add_vinsn (av_set_t *, vinsn_t, int, int); -extern av_set_t av_set_copy (av_set_t, int); +extern rhs_t av_set_lookup (av_set_t, vinsn_t); +extern rhs_t av_set_lookup_other_equiv_rhs (av_set_t, vinsn_t); +extern bool av_set_is_in_p (av_set_t, vinsn_t); +extern av_set_t av_set_copy (av_set_t); extern void av_set_union_and_clear (av_set_t *, av_set_t *); extern void av_set_clear (av_set_t *); extern void av_set_leave_one (av_set_t *); extern rhs_t av_set_element (av_set_t, int); -extern rhs_t av_set_lookup_insn (av_set_t, insn_t); extern void av_set_substract_cond_branches (av_set_t *); extern void av_set_intersect (av_set_t *, av_set_t); -extern void av_set_add_insn (av_set_t *, insn_t); + +extern void sel_save_haifa_priorities (void); + +extern void sel_init_global_and_expr (bb_vec_t); +extern void sel_finish_global_and_expr (void); /* Dependence analysis functions. */ -extern bool has_dependence_p (rhs_t, insn_t); -extern bool tick_check_p (vinsn_t, deps_t, fence_t); +extern void sel_clear_has_dependence (void); +extern ds_t has_dependence_p (rhs_t, insn_t, ds_t **); + +extern bool tick_check_p (rhs_t, deps_t, fence_t); /* Functions to work with insns. */ -extern bool lhs_equals_reg_p (insn_t, rtx); -extern bool bb_header_p (insn_t); -extern bool insn_valid_p (insn_t); +extern bool lhs_of_insn_equals_to_reg_p (insn_t, rtx); +extern bool insn_rtx_valid (rtx); extern bool insn_eligible_for_subst_p (insn_t); extern void get_dest_and_mode (rtx, rtx *, enum machine_mode *); + +extern void sel_init_new_insns (void); +extern void sel_finish_new_insns (void); + extern bool bookkeeping_can_be_created_if_moved_through_p (insn_t); extern insn_t copy_insn_out_of_stream (vinsn_t); extern insn_t copy_insn_and_insert_before (insn_t, insn_t); extern void sched_sel_remove_insn (insn_t); extern void transfer_data_sets (insn_t, insn_t); -extern int dfa_cost (insn_t, fence_t); -extern void release_lv_set_for_insn (rtx); +extern int vinsn_dfa_cost (vinsn_t, fence_t); extern bool bb_header_p (insn_t); /* Basic block and CFG functions. */ -extern insn_t bb_head (basic_block); + +extern insn_t sel_bb_header (basic_block); +extern bool sel_bb_header_p (insn_t); +extern bool sel_bb_empty_p_1 (basic_block, bool); +extern bool sel_bb_empty_p (basic_block); +extern insn_t sel_bb_end (basic_block); +extern bool sel_bb_end_p (insn_t); + extern bool in_current_region_p (basic_block); -extern bool bb_empty_p (basic_block); + +extern void sel_init_bbs (bb_vec_t, basic_block); +extern void sel_finish_bbs (void); + extern int cfg_succs_n (insn_t, int); +extern bool sel_insn_has_single_succ_p (insn_t, int); extern void cfg_succs_1 (insn_t, int, insn_t **, int *); extern void cfg_succs (insn_t, insn_t **, int *); extern insn_t cfg_succ_1 (insn_t, int); extern insn_t cfg_succ (insn_t); extern bool num_preds_gt_1 (insn_t); + +extern bool is_ineligible_successor (insn_t, ilist_t); + extern bool bb_ends_ebb_p (basic_block); extern bool in_same_ebb_p (insn_t, insn_t); -extern bool is_ineligible_successor (insn_t, ilist_t); -extern void sel_add_or_remove_bb_1 (basic_block, bool); -extern void sel_add_or_remove_bb (basic_block, bool); -extern void sel_remove_empty_bb (basic_block, bool); + +extern basic_block sel_create_basic_block (void *, void *, basic_block); + +extern void sel_add_or_remove_bb (basic_block, int); +extern basic_block sel_create_basic_block_before (basic_block); +extern void sel_remove_empty_bb (basic_block, bool, bool); extern basic_block sel_split_block (basic_block, insn_t); extern basic_block sel_split_edge (edge); +extern void sel_merge_blocks (basic_block, basic_block); extern basic_block sel_redirect_edge_force (edge, basic_block); extern edge sel_redirect_edge_and_branch (edge, basic_block); -extern basic_block sel_create_basic_block (void *, void *, basic_block); -extern basic_block sel_create_basic_block_before (basic_block); extern void pipeline_outer_loops (void); extern void pipeline_outer_loops_init (void); extern void pipeline_outer_loops_finish (void); @@ -898,15 +926,24 @@ extern loop_p get_loop_nest_for_rgn (unsigned int); extern bool considered_for_pipelining_p (struct loop *); extern void make_region_from_loop_preheader (VEC(basic_block, heap) **); extern void sel_add_loop_preheader (void); +extern bool sel_is_loop_preheader_p (basic_block); extern void clear_outdated_rtx_info (basic_block); +/* Expression transformation routines. */ +extern rtx create_insn_rtx_from_pattern (rtx); +extern vinsn_t create_vinsn_from_insn_rtx (rtx); +extern rtx create_copy_of_insn_rtx (rtx); +extern void change_vinsn_in_expr (expr_t, vinsn_t); /* Various initialization functions. */ extern void init_lv_sets (void); extern void free_lv_sets (void); extern void setup_nop_and_exit_insns (void); -extern void free_exit_insn_data (void); -extern void setup_sched_and_deps_infos (void); +extern void free_nop_and_exit_insns (void); +extern void setup_empty_vinsn (void); +extern void free_empty_vinsn (void); +extern void sel_setup_common_sched_info (void); +extern void sel_setup_sched_infos (void); /* Successor iterator backend. */ @@ -997,7 +1034,7 @@ get_all_loop_exits (basic_block bb) /* If bb is empty, and we're skipping to loop exits, then consider bb as a possible gate to the inner loop now. */ - while (bb_empty_p (bb) + while (sel_bb_empty_p (bb) && in_current_region_p (bb)) bb = single_succ (bb); @@ -1234,7 +1271,7 @@ _eligible_successor_edge_p (edge e1, basic_block real_pred, int flags, edge *e2p /* Skip empty blocks, but be careful not to leave the region. */ while (1) { - if (!bb_empty_p (bb)) + if (!sel_bb_empty_p_1 (bb, false)) break; if (!in_current_region_p (bb) @@ -1290,6 +1327,8 @@ _eligible_successor_edge_p (edge e1, basic_block real_pred, int flags, edge *e2p #define FOR_EACH_SUCC(SUCC, ITER, INSN) \ FOR_EACH_SUCC_1 (SUCC, ITER, INSN, SUCCS_NORMAL) +extern regset sel_all_regs; + /* Return the next block of BB not running into inconsistencies. */ static inline basic_block bb_next_bb (basic_block bb) diff --git a/gcc/sel-sched.c b/gcc/sel-sched.c index ff0d74c971e..ae355fa2b5f 100644 --- a/gcc/sel-sched.c +++ b/gcc/sel-sched.c @@ -81,6 +81,10 @@ bool pipelining_p; /* True if bookkeeping is enabled. */ bool bookkeeping_p; + +/* True if we should make an aditional pass to set somewhat correct + sched cycles. */ +bool reset_sched_cycles_p; /* Definitions of local types and macros. */ @@ -149,8 +153,8 @@ bitmap_head *forced_ebb_heads = &_forced_ebb_heads; /* True when the first lv set should be ignored when updating liveness. */ static bool ignore_first = false; -/* Holds the length of the ready list. */ -static int old_ready_veclen = 0; +/* Number of insns max_issue has initialized data structures for. */ +static int max_issue_size = 0; /* Whether do we need a stall currently. */ static int need_stall = 0; @@ -175,10 +179,10 @@ static bool rtx_search (rtx, rtx); static int sel_rank_for_schedule (const void *, const void *); static bool equal_after_moveup_path_p (rhs_t, ilist_t, rhs_t); static regset compute_live (insn_t); -static basic_block generate_bookkeeping_insn (rhs_t, rtx, insn_t, edge, edge); +static basic_block generate_bookkeeping_insn (rhs_t, insn_t, edge, edge); static bool find_used_regs (insn_t, av_set_t, regset, HARD_REG_SET *, def_list_t *); -static bool move_op (insn_t, av_set_t, rtx, ilist_t, edge, edge, rhs_t); +static bool move_op (insn_t, av_set_t, ilist_t, edge, edge, rhs_t); static void sel_sched_region_1 (void); static void sel_sched_region_2 (sel_sched_region_2_data_t); @@ -257,36 +261,45 @@ extract_new_fences_from (fence_t fence, flist_tail_t new_fences, be not performed because it's either RHS' vinsn doesn't contain INSN's destination or the resulting insn is invalid for the target machine. */ static bool -substitute_rhs (rhs_t rhs, insn_t insn) +substitute_reg_in_rhs (rhs_t rhs, insn_t insn) { rtx *where; bool new_insn_valid; vinsn_t *vi = &RHS_VINSN (rhs); + if (VINSN_UNIQUE_P (*vi)) + /* Unique vinsns can't play this game. */ + return false; + /* Do not try to replace in SET_DEST. Although we'll choose new register for the RHS, we don't want to change RHS' original reg. If the insn is not SET, we may still be able to substitute something in it, and if we're here (don't have deps), it doesn't write INSN's dest. */ - where = VINSN_HAS_RHS (*vi) ? &VINSN_RHS (*vi) : - &PATTERN (VINSN_INSN (*vi)); + where = (VINSN_SEPARABLE_P (*vi) + ? &VINSN_RHS (*vi) + : &PATTERN (VINSN_INSN (*vi))); /* Substitute if INSN has a form of x:=y and LHS(INSN) occurs in *VI. */ - if (insn_eligible_for_subst_p (insn) && !VINSN_UNIQUE_P (*vi) - && rtx_search (INSN_LHS (insn), *where)) + if (insn_eligible_for_subst_p (insn) && rtx_search (INSN_LHS (insn), *where)) { - /* We should copy these rtxes before substitution. */ - insn_t new_insn = copy_insn_out_of_stream (*vi); + rtx new_insn; + rtx *where_replace; + + /* We should copy these rtxes before substitution. */ + new_insn = create_copy_of_insn_rtx (VINSN_INSN_RTX (*vi)); /* Where we'll replace. */ /* WHERE_REPLACE should point inside NEW_INSN, so INSN_RHS couldn't be used instead of SET_SRC. */ - rtx *where_replace = INSN_RHS (new_insn) ? - &SET_SRC (PATTERN (new_insn)) : &PATTERN (new_insn); + where_replace = (VINSN_SEPARABLE_P (*vi) + ? &SET_SRC (PATTERN (new_insn)) + : &PATTERN (new_insn)); new_insn_valid = validate_replace_rtx_part (INSN_LHS (insn), - expr_copy (INSN_RHS (insn)), + copy_rtx (INSN_RHS (insn)), where_replace, new_insn); + /* ??? Actually, constrain_operands result depends upon choice of destination register. E.g. if we allow single register to be an rhs, and if we try to move dx=ax(as rhs) through ax=dx, we'll result @@ -295,19 +308,16 @@ substitute_rhs (rhs_t rhs, insn_t insn) leaving it for now. */ if (new_insn_valid) { - gcc_assert (insn_valid_p (new_insn)); - - vinsn_detach (*vi); - *vi = INSN_VI (new_insn); + if (!insn_rtx_valid (new_insn)) + gcc_unreachable (); - /* Recompute INSN_COST. */ - INSN_COST (new_insn) = insn_cost (new_insn, 0, 0); + change_vinsn_in_expr (rhs, create_vinsn_from_insn_rtx (new_insn)); return true; } else { - sched_sel_remove_insn (new_insn); + /* It would nice to free new_insn or return it to some pool. */ return false; } } @@ -348,8 +358,8 @@ replace_in_vinsn_using_bitmask_1 (rtx *cur_rtx, void *arg) index K in bitmap mask means that the K-th occurence of WHAT will be replaced by BY_WHAT in WHERE, "0" means that it won't be changed. */ static void -replace_in_vinsn_using_bitmask (vinsn_t where, rtx what, rtx by_what, - unsigned long bitmask) +replace_in_rtx_using_bitmask (rtx *wherep, rtx what, rtx by_what, + unsigned long bitmask) { struct rtx_search_arg arg; @@ -359,15 +369,11 @@ replace_in_vinsn_using_bitmask (vinsn_t where, rtx what, rtx by_what, arg.y = by_what; arg.bitmask = bitmask; - for_each_rtx (VINSN_SEPARABLE (where) ? - &VINSN_RHS (where) : &VINSN_PATTERN (where), + for_each_rtx (wherep, &replace_in_vinsn_using_bitmask_1, (void *) &arg); /* Make sure all occurences has been replaced. */ gcc_assert (!arg.bitmask); - - /* Recompute VINSN_LHS and VINSN_RHS. */ - recompute_vinsn_lhs_rhs (where); } /* Helper function for count_occurences_equiv. */ @@ -482,9 +488,14 @@ un_substitute (av_set_t *av_ptr, rtx insn) { av_set_iterator av_iter; rhs_t rhs; - av_set_t new_set = NULL; + av_set_t new_set; rtx pat, src_reg, dst_reg; + if (!flag_sel_sched_substitution) + return; + + new_set = NULL; + pat = PATTERN (insn); @@ -521,12 +532,16 @@ un_substitute (av_set_t *av_ptr, rtx insn) FOR_EACH_RHS (rhs, av_iter, *av_ptr) { - rtx where_count = RHS_SCHEDULE_AS_RHS (rhs) ? - VINSN_RHS (RHS_VINSN (rhs)) : RHS_PAT (rhs); - - unsigned n_occur = count_occurrences_equiv (src_reg, where_count, 0); + vinsn_t vi = RHS_VINSN (rhs); + rtx where_count; + unsigned n_occur; unsigned long bitmask; + where_count = (VINSN_SEPARABLE_P (vi) ? + VINSN_RHS (vi) : VINSN_PATTERN (vi)); + + n_occur = count_occurrences_equiv (src_reg, where_count, 0); + /* Try next rhs, if no occurences. */ if (!n_occur) continue; @@ -541,24 +556,36 @@ un_substitute (av_set_t *av_ptr, rtx insn) bitmask == 0 in the original set. */ for (bitmask = 1; bitmask < ((unsigned long)1 << n_occur); bitmask++) { - vinsn_t vi; + vinsn_t vi = RHS_VINSN (rhs); + expr_def _tmp_rhs, *tmp_rhs = &_tmp_rhs; + rtx new_insn; + rtx *wherep; - insn_t new_insn = copy_insn_out_of_stream (RHS_VINSN (rhs)); - replace_in_vinsn_using_bitmask (INSN_VI (new_insn), src_reg, - dst_reg, bitmask); + new_insn = create_copy_of_insn_rtx (VINSN_INSN (vi)); - if (!insn_valid_p (new_insn)) + wherep = (VINSN_SEPARABLE_P (vi) + ? &SET_SRC (PATTERN (new_insn)) + : &PATTERN (new_insn)); + + replace_in_rtx_using_bitmask (wherep, src_reg, dst_reg, bitmask); + + if (!insn_rtx_valid (new_insn)) { - sched_sel_remove_insn (new_insn); + /* It would nice to free new_insn or return it to some pool. */ continue; } - - vi = INSN_VI (new_insn); - gcc_assert (VINSN_SEPARABLE (vi) == RHS_SCHEDULE_AS_RHS (rhs)); - av_set_add_vinsn (&new_set, vi, RHS_SPEC (rhs), RHS_PRIORITY (rhs)); - /* VI should be deallocated with the av_set. */ - vinsn_detach (vi); - gcc_assert (VINSN_COUNT (vi) == 1); + + vi = create_vinsn_from_insn_rtx (new_insn); + + gcc_assert (VINSN_SEPARABLE_P (vi) + == EXPR_SEPARABLE_P (rhs)); + + copy_expr (tmp_rhs, rhs); + change_vinsn_in_expr (tmp_rhs, vi); + + av_set_add (&new_set, tmp_rhs); + + clear_expr (tmp_rhs); } } av_set_union_and_clear (av_ptr, &new_set); @@ -567,40 +594,21 @@ un_substitute (av_set_t *av_ptr, rtx insn) /* Functions to support register renaming. */ -/* Substitute VI's set source with NEW_SRC_REG. Returns newly created insn - that has NEW_SRC_REG as its source with correct VINSN_LHS, VINSN_RHS, deps - and other structures. */ -static insn_t -replace_src_with_reg (vinsn_t vi, rtx new_src_reg) +/* Substitute VI's set source with REGNO. Returns newly created pattern + that has REGNO as its source. */ +static rtx +create_insn_rtx_with_rhs (vinsn_t vi, rtx rhs_rtx) { - insn_t insn1, insn2; - insn_t orig_insn = VINSN_INSN (vi); - int res; - - /* We should deal here only with non-unique insns. */ - gcc_assert (!VINSN_UNIQUE_P (vi)); - gcc_assert (VINSN_TYPE (vi) == SET); - - /* Copy rtx before substitution. */ - insn1 = copy_insn_out_of_stream (vi); - - /* Replace it. */ - res = validate_change (insn1, &SET_SRC (PATTERN (insn1)), new_src_reg, 0); - gcc_assert (res); + rtx lhs_rtx; + rtx pattern; + rtx insn_rtx; - /* Copy it once again to generate correct deps and lhs & rhs data. */ - insn2 = copy_insn_and_insert_before (insn1, VINSN_INSN (vi)); + lhs_rtx = copy_rtx (VINSN_LHS (vi)); - /* Remove temp insn. */ - sched_sel_remove_insn (insn1); + pattern = gen_rtx_SET (VOIDmode, lhs_rtx, rhs_rtx); + insn_rtx = create_insn_rtx_from_pattern (pattern); - gcc_assert (insn_valid_p (insn2)); - - INSN_SEQNO (insn2) = INSN_SEQNO (orig_insn); - INSN_PRIORITY (insn2) = INSN_PRIORITY (orig_insn); - INSN_COST (insn2) = insn_cost (insn2, 0, 0); - - return insn2; + return insn_rtx; } /* Returns whether INSN's src can be replaced with register number @@ -632,8 +640,7 @@ replace_src_with_reg_ok_p (insn_t insn, rtx new_src_reg) rtx dst_loc; bool res; - gcc_assert (!VINSN_UNIQUE_P (vi)); - gcc_assert (VINSN_TYPE (vi) == SET); + gcc_assert (VINSN_SEPARABLE_P (vi)); get_dest_and_mode (insn, &dst_loc, &mode); gcc_assert (mode == GET_MODE (new_src_reg)); @@ -649,58 +656,6 @@ replace_src_with_reg_ok_p (insn_t insn, rtx new_src_reg) return res; } -/* Substitute SET_DEST in the given vinsn VI for the register NEW_REG. - If CLONE_P is true, make a copy of it (assuming it's a hard reg.) */ -static insn_t -replace_dest_with_reg_in_vinsn (vinsn_t vi, rtx new_reg, bool clone_p) -{ - insn_t new_insn, tmp_insn; - int res; - - /* We should deal here only with non-unique insns. */ - gcc_assert (!VINSN_UNIQUE_P (vi)); - gcc_assert (VINSN_TYPE (vi) == SET); - gcc_assert (GET_MODE (VINSN_LHS (vi)) == GET_MODE (new_reg)); - - /* Copy rtx before substitution. */ - tmp_insn = copy_insn_out_of_stream (vi); - - /* Replace it. */ - if (clone_p && HARD_REGISTER_P (new_reg)) - new_reg = copy_rtx (new_reg); - res = validate_change (tmp_insn, &SET_DEST (PATTERN (tmp_insn)), - new_reg, 0); - gcc_assert (res); - - /* Copy it once again to generate correct deps and lhs & rhs data. */ - new_insn = copy_insn_out_of_stream (INSN_VI (tmp_insn)); - - /* FIXME: Can't remove insn because of shared rtx'es. - This should be fixed somehow. */ - sched_sel_remove_insn (tmp_insn); - - INSN_COST (new_insn) = INSN_COST (VINSN_INSN (vi)); - - gcc_assert (insn_valid_p (new_insn)); - - return new_insn; -} - -/* Substitute SET_DEST in the given rhs R for the register NEW_REG. - Also generate new vinsn. SET_DEST may be arbitrary rtx, not only - register. */ -static void -replace_dest_with_reg_in_rhs (rhs_t r, rtx new_reg) -{ - vinsn_t vi = RHS_VINSN (r); - vinsn_t vi1; - - vi1 = INSN_VI (replace_dest_with_reg_in_vinsn (vi, new_reg, false)); - vinsn_detach (vi); - - RHS_VINSN (r) = vi1; -} - /* Returns whether INSN still be valid after replacing it's DEST with register NEW_REG. */ static bool @@ -709,9 +664,8 @@ replace_dest_with_reg_ok_p (insn_t insn, rtx new_reg) vinsn_t vi = INSN_VI (insn); bool res; - /* We should deal here only with non-unique insns. */ - gcc_assert (!VINSN_UNIQUE_P (vi)); - gcc_assert (VINSN_TYPE (vi) == SET); + /* We should deal here only with separable insns. */ + gcc_assert (VINSN_SEPARABLE_P (vi)); gcc_assert (GET_MODE (VINSN_LHS (vi)) == GET_MODE (new_reg)); /* See whether SET_DEST can be replaced with this register. */ @@ -722,13 +676,55 @@ replace_dest_with_reg_ok_p (insn_t insn, rtx new_reg) return res; } +/* Create a pattern with rhs of VI and lhs of LHS_RTX. */ +static rtx +create_insn_rtx_with_lhs (vinsn_t vi, rtx lhs_rtx) +{ + rtx rhs_rtx; + rtx pattern; + rtx insn_rtx; + + rhs_rtx = copy_rtx (VINSN_RHS (vi)); + + pattern = gen_rtx_SET (VOIDmode, lhs_rtx, rhs_rtx); + insn_rtx = create_insn_rtx_from_pattern (pattern); + + return insn_rtx; +} + +/* Substitute lhs in the given expression RHS for the register with number + NEW_REGNO. SET_DEST may be arbitrary rtx, not only register. */ +static void +replace_dest_with_reg_in_rhs (rhs_t rhs, rtx new_reg) +{ + rtx insn_rtx; + vinsn_t vinsn; + + insn_rtx = create_insn_rtx_with_lhs (RHS_VINSN (rhs), new_reg); + vinsn = create_vinsn_from_insn_rtx (insn_rtx); + + change_vinsn_in_expr (rhs, vinsn); +} + +/* Return a destination register, if any, of EXPR. */ +static rtx +expr_dest_reg (expr_t expr) +{ + rtx dest = VINSN_LHS (RHS_VINSN (expr)); + + if (dest != NULL_RTX && REG_P (dest)) + return dest; + + return NULL_RTX; +} + /* Returns the REGNO of the R's destination. */ static unsigned rhs_dest_regno (rhs_t r) { - rtx dest = VINSN_LHS (RHS_VINSN (r)); + rtx dest = expr_dest_reg (r); - gcc_assert (dest && REG_P (dest)); + gcc_assert (dest != NULL_RTX); return REGNO (dest); } @@ -936,8 +932,7 @@ mark_unavailable_hard_regs (def_t def, HARD_REG_SET *unavailable_hard_regs, /* FIXME: ? this loop can be optimized if traversing union of regs_ever_live and call_used_regs? */ - || (! regs_ever_live[cur_reg + i] - && ! call_used_regs[cur_reg + i]) + || !REGNO_REG_SET_P (sel_all_regs, (cur_reg + i)) #ifdef LEAF_REGISTERS /* We can't use a non-leaf register if we're in a leaf function. */ @@ -1016,9 +1011,9 @@ static int reg_rename_tick[FIRST_PSEUDO_REGISTER]; tick is returned to achieve more even register allocation. If no register satisfies the above conditions, NULL_RTX is returned. */ - static rtx -choose_best_reg (HARD_REG_SET unavailable, def_list_t original_insns) +choose_best_reg_1 (HARD_REG_SET unavailable, def_list_t original_insns, + bool *is_orig_reg_p_ptr) { int best_new_reg; int cur_reg; @@ -1043,6 +1038,8 @@ choose_best_reg (HARD_REG_SET unavailable, def_list_t original_insns) } /* If original register is available, return it. */ + *is_orig_reg_p_ptr = true; + FOR_EACH_DEF (def, i, original_insns) { rtx orig_dest = SET_DEST (PATTERN (def->orig_insn)); @@ -1060,21 +1057,25 @@ choose_best_reg (HARD_REG_SET unavailable, def_list_t original_insns) if (!TEST_HARD_REG_BIT (unavailable, REGNO (orig_dest))) { gcc_assert (mode != VOIDmode); + /* Hard registers should not be shared. */ return gen_rtx_REG (mode, REGNO (orig_dest)); } } + *is_orig_reg_p_ptr = false; + best_new_reg = -1; /* Among all available regs choose the register that was allocated earliest. */ for (cur_reg = 0; cur_reg < FIRST_PSEUDO_REGISTER; cur_reg++) - if (!TEST_HARD_REG_BIT (unavailable, cur_reg)) { - if (best_new_reg < 0 - ||reg_rename_tick[cur_reg] < reg_rename_tick[best_new_reg]) - best_new_reg = cur_reg; - } + if (!TEST_HARD_REG_BIT (unavailable, cur_reg)) + { + if (best_new_reg < 0 + ||reg_rename_tick[cur_reg] < reg_rename_tick[best_new_reg]) + best_new_reg = cur_reg; + } if (best_new_reg >= 0) { @@ -1086,6 +1087,27 @@ choose_best_reg (HARD_REG_SET unavailable, def_list_t original_insns) return NULL_RTX; } +/* A regset that includes all registers that present in current region. + Used in assertions to check that we don't introduce new registers when + should not. */ +static regset_head _sel_all_regs; +regset sel_all_regs = &_sel_all_regs; + +/* A wrapper around choose_best_reg_1 () to verify that we make correct + assumptions about available registers in the function. */ +static rtx +choose_best_reg (HARD_REG_SET unavailable, def_list_t original_insns, + bool *is_orig_reg_p_ptr) +{ + rtx best_reg = choose_best_reg_1 (unavailable, original_insns, + is_orig_reg_p_ptr); + + gcc_assert (best_reg == NULL_RTX + || REGNO_REG_SET_P (sel_all_regs, REGNO (best_reg))); + + return best_reg; +} + /* Choose the pseudo register for storing rhs value. As this is supposed to work before reload, we return either the original register or make the new one. If we work with hard regs, check also UNAVAILABLE_HARD_REGS. @@ -1096,7 +1118,7 @@ choose_best_reg (HARD_REG_SET unavailable, def_list_t original_insns) static rtx choose_best_pseudo_reg (regset used_regs, HARD_REG_SET unavailable_hard_regs, - def_list_t original_insns) + def_list_t original_insns, bool *is_orig_reg_p_ptr) { def_list_iterator i; def_t def; @@ -1105,8 +1127,10 @@ choose_best_pseudo_reg (regset used_regs, /* We should not use this after reload. */ gcc_assert (!reload_completed); - + /* If original register is available, return it. */ + *is_orig_reg_p_ptr = true; + FOR_EACH_DEF (def, i, original_insns) { rtx dest = SET_DEST (PATTERN (def->orig_insn)); @@ -1138,9 +1162,11 @@ choose_best_pseudo_reg (regset used_regs, return dest; } } + + *is_orig_reg_p_ptr = false; - /* We had some original hard registers that couldn't be used. Those were likely - special. Don't try to create a pseudo. */ + /* We had some original hard registers that couldn't be used. + Those were likely special. Don't try to create a pseudo. */ if (bad_hard_regs) return NULL_RTX; @@ -1156,7 +1182,7 @@ choose_best_pseudo_reg (regset used_regs, register; - RHS_SCHEDULE_AS_RHS is false but the insn sets/clobbers one of the registers that are used on the moving path. */ -static rtx +static bool find_best_reg_for_rhs (rhs_t rhs, blist_t bnds) { av_set_iterator i2; @@ -1165,8 +1191,10 @@ find_best_reg_for_rhs (rhs_t rhs, blist_t bnds) HARD_REG_SET hard_regs_used, unavailable_hard_regs; rtx best_reg = NULL_RTX; blist_t bnds1 = bnds; - ilist_t original_insns = NULL; - bool res = 0; + def_list_t original_insns = NULL; + int res = 0; + bool reg_ok = true; + bool is_orig_reg_p = false; used_regs = get_clear_regset_from_pool (); CLEAR_HARD_REG_SET (unavailable_hard_regs); @@ -1179,7 +1207,7 @@ find_best_reg_for_rhs (rhs_t rhs, blist_t bnds) /* If the chosen best rhs doesn't belong to current boundary, skip it. */ - if (!av_set_is_in_p (BND_AV1 (bnd), rhs)) + if (!av_set_is_in_p (BND_AV1 (bnd), RHS_VINSN (rhs))) continue; /* Put in ORIG_OPS all rhses from this boundary that became @@ -1189,9 +1217,7 @@ find_best_reg_for_rhs (rhs_t rhs, blist_t bnds) ilist_t root = enable_moveup_set_path_p ? BND_PTR (bnd) : NULL; if (equal_after_moveup_path_p (rhs_orig, root, rhs)) - av_set_add_vinsn (&orig_ops, RHS_VINSN (rhs_orig), - RHS_SPEC (rhs_orig), - RHS_PRIORITY (rhs_orig)); + av_set_add (&orig_ops, rhs_orig); } /* Compute used regs and OR it into the USED_REGS. */ @@ -1213,6 +1239,7 @@ find_best_reg_for_rhs (rhs_t rhs, blist_t bnds) } else { + gcc_unreachable (); print ("Unable to find original op with find_used_regs."); break; } @@ -1231,7 +1258,7 @@ find_best_reg_for_rhs (rhs_t rhs, blist_t bnds) } #endif - if (RHS_SCHEDULE_AS_RHS (rhs)) + if (EXPR_SEPARABLE_P (rhs)) { /* Turn everything in hard regs after reload. */ if (reload_completed) @@ -1240,16 +1267,21 @@ find_best_reg_for_rhs (rhs_t rhs, blist_t bnds) /* Join hard registers unavailable due to register class restrictions and live range intersection. */ IOR_HARD_REG_SET (hard_regs_used, unavailable_hard_regs); - best_reg = choose_best_reg (hard_regs_used, original_insns); + best_reg = choose_best_reg (hard_regs_used, original_insns, + &is_orig_reg_p); } else best_reg = choose_best_pseudo_reg (used_regs, unavailable_hard_regs, - original_insns); - - /* Try whether we'll be able to generate the insn 'dest := best_reg' - at the place of the original operation. */ - if (best_reg) + original_insns, + &is_orig_reg_p); + + if (!is_orig_reg_p && sel_vinsn_cost (EXPR_VINSN (rhs)) < 2) + best_reg = NULL_RTX; + + if (best_reg != NULL_RTX) + /* Try whether we'll be able to generate the insn + 'dest := best_reg' at the place of the original operation. */ { ilist_t p = original_insns; @@ -1259,22 +1291,41 @@ find_best_reg_for_rhs (rhs_t rhs, blist_t bnds) { def_t def = DEF_LIST_DEF (p); - gcc_assert (VINSN_HAS_RHS (INSN_VI (def->orig_insn))); + gcc_assert (EXPR_SEPARABLE_P (INSN_EXPR (def->orig_insn))); if (!replace_src_with_reg_ok_p (def->orig_insn, best_reg) || !replace_dest_with_reg_ok_p (def->orig_insn, - best_reg)) + best_reg)) { /* Insn will be removed from rhs_vliw below. */ /* FIXME: may be it will work with other regs? Not with example above, though - we can't figure that the only option is to generate subreg. */ - best_reg = NULL_RTX; + reg_ok = false; break; } + p = ILIST_NEXT (p); + } + + if (reg_ok) + { + RGN_HAS_RENAMING_P (CONTAINING_RGN (BB_TO_BLOCK (0))) = 1; + + /* Make sure that RHS has the right destination + register. */ + if (rhs_dest_regno (rhs) != REGNO (best_reg)) + { + replace_dest_with_reg_in_rhs (rhs, best_reg); + + /* The resulting insn should be valid. */ + if (!insn_rtx_valid (RHS_INSN (rhs))) + gcc_unreachable (); + } } } + else + reg_ok = false; } else { @@ -1282,25 +1333,204 @@ find_best_reg_for_rhs (rhs_t rhs, blist_t bnds) any of the HARD_REGS_USED set. */ if (vinsn_writes_one_of_regs_p (RHS_VINSN (rhs), used_regs, unavailable_hard_regs)) - best_reg = NULL_RTX; + reg_ok = false; else { - /* Any non-negative number. - ??? Introduce another parameter instead of this. */ - best_reg = (rtx) 1; + gcc_assert (reg_ok); + best_reg = NULL_RTX; } } } else - best_reg = NULL_RTX; + reg_ok = false; ilist_clear (&original_insns); return_regset_to_pool (used_regs); - return best_reg; + return reg_ok; } +/* Flag to enable / disable ia64 speculation. */ +static bool sel_speculation_p = true; + +/* Return true if dependence described by DS can be overcomed. */ +static bool +can_overcome_dep_p (ds_t ds) +{ + if (spec_info == NULL || !sel_speculation_p) + return false; + + /* Leave only speculative data. */ + ds &= SPECULATIVE; + + if (ds == 0) + return false; + + { + /* !!! FIXME: + 1. Make sched-deps.c produce only those non-hard dependencies, + that we can overcome. + 2. Introduce several machine flags to turn on/off different kinds of + speculation. */ + ds_t spec_mask = spec_info->mask; + + if ((ds & spec_mask) != ds) + return false; + } + + if (ds_weak (ds) < spec_info->weakness_cutoff) + return false; + + return true; +} + +/* Get a speculation check instruction from the target. SPEC_EXPR is a + speculative expression. */ +static rtx +create_speculation_check_insn_rtx (rtx spec_insn_rtx, ds_t check_ds) +{ + rtx check_pattern; + + check_pattern = targetm.sched.gen_spec_check (spec_insn_rtx, NULL_RTX, + check_ds); + + gcc_assert (check_pattern != NULL); + + return create_insn_rtx_from_pattern (check_pattern); +} + +/* Try to transform EXPR to data speculative version. Return true on + success. */ +static bool +apply_spec_to_expr (expr_t expr, ds_t ds) +{ + int res; + rtx orig_insn_rtx; + rtx spec_pat; + + orig_insn_rtx = EXPR_INSN_RTX (expr); + + res = sched_speculate_insn (orig_insn_rtx, ds, &spec_pat); + + switch (res) + { + case 0: + spec_pat = copy_rtx (PATTERN (orig_insn_rtx)); + + /* FALLTHRU */ + + case 1: + { + rtx spec_insn_rtx = create_insn_rtx_from_pattern (spec_pat); + vinsn_t spec_vinsn = create_vinsn_from_insn_rtx (spec_insn_rtx); + + change_vinsn_in_expr (expr, spec_vinsn); + EXPR_SPEC_DONE_DS (expr) = ds; + + return true; + } + + case -1: + return false; + + default: + gcc_unreachable (); + return false; + } +} + +/* Try to make EXPR speculative and on success remove from DSP those + dependencies that were overcame. Return true on success. */ +static bool +speculate_expr (expr_t expr, ds_t ds) +{ + ds_t target_ds = (ds & SPECULATIVE); + ds_t current_ds = EXPR_SPEC_DONE_DS (expr); + ds_t combined_ds = ds_full_merge (current_ds, target_ds, NULL_RTX, NULL_RTX); + + if (apply_spec_to_expr (expr, combined_ds)) + /* We already have all necessary speculations. */ + return true; + + return false; +} + +/* Return true if there is a speculative dependence between INSN and EXPR. */ +static ds_t +has_spec_dependence_p (expr_t expr, insn_t insn) +{ + ds_t *has_dep_p; + ds_t full_ds; + + full_ds = has_dependence_p (expr, insn, &has_dep_p); + + if (full_ds == 0) + return 0; + + if (EXPR_SEPARABLE_P (expr) + && can_overcome_dep_p (has_dep_p [DEPS_IN_RHS])) + return has_dep_p [DEPS_IN_RHS]; + + if (can_overcome_dep_p (full_ds)) + return full_ds; + + return 0; +} + +/* Add to AVP those exprs that might have been transformed to their speculative + versions when moved through INSN. */ +static void +un_speculate (av_set_t *avp, insn_t insn) +{ + av_set_iterator av_it; + expr_t expr; + av_set_t new_av; + + if (spec_info == NULL || !sel_speculation_p) + return; + + new_av = NULL; + + FOR_EACH_RHS (expr, av_it, *avp) + { + ds_t expr_spec_done_ds = EXPR_SPEC_DONE_DS (expr); + ds_t full_ds; + + if (expr_spec_done_ds == 0) + continue; + + full_ds = has_spec_dependence_p (expr, insn); + + if (full_ds == 0) + continue; + + { + expr_def _new_expr, *new_expr = &_new_expr; + + copy_expr (new_expr, expr); + + { + bool b; + + full_ds = ds_get_speculation_types (full_ds); + expr_spec_done_ds &= ~full_ds; + + b = apply_spec_to_expr (new_expr, expr_spec_done_ds); + gcc_assert (b); + + EXPR_SPEC_TO_CHECK_DS (new_expr) |= full_ds; + } + + av_set_add (&new_av, new_expr); + + clear_expr (new_expr); + } + } + + av_set_union_and_clear (avp, &new_av); +} + /* Moveup_* helpers for code motion and computing av sets. */ /* Modifies INSN_TO_MOVE_UP so it can be moved through the THROUGH_INSN, @@ -1310,12 +1540,8 @@ moveup_rhs (rhs_t insn_to_move_up, insn_t through_insn) { vinsn_t vi = RHS_VINSN (insn_to_move_up); insn_t insn = VINSN_INSN (vi); - enum MOVEUP_RHS_CODE rcode; - - /* Don't move trapping insns through jumps. */ - if (MAY_TRAP (insn) - && control_flow_insn_p (through_insn)) - return MOVEUP_RHS_NULL; + ds_t *has_dep_p; + ds_t full_ds; if (VINSN_UNIQUE_P (vi)) { @@ -1329,6 +1555,15 @@ moveup_rhs (rhs_t insn_to_move_up, insn_t through_insn) && BLOCK_FOR_INSN (through_insn) != BLOCK_FOR_INSN (insn)) /* Don't move what we can't move. */ return MOVEUP_RHS_NULL; + + if (SCHED_GROUP_P (insn)) + /* Don't move SCHED_GROUP instruction through anything. + If we don't force this, then it will be possible to start + scheduling a sched_group before all its dependencies are + resolved. + ??? Haifa deals with this issue by delaying the SCHED_GROUP + as late as possible through rank_for_schedule (). */ + return MOVEUP_RHS_NULL; } else { @@ -1336,21 +1571,40 @@ moveup_rhs (rhs_t insn_to_move_up, insn_t through_insn) if (!bookkeeping_p && bookkeeping_can_be_created_if_moved_through_p (through_insn)) + /* If bookkeeping is disabled don't try to move those insns that can + possible generate one. */ return MOVEUP_RHS_NULL; } - if (!has_dependence_p (insn_to_move_up, through_insn)) + full_ds = has_dependence_p (insn_to_move_up, through_insn, &has_dep_p); + + if (full_ds == 0) return MOVEUP_RHS_SAME; else { - /* We can move UNIQUE insn up only as a whole and unchanged, - so it shouldn't have any dependencies. */ - if (VINSN_UNIQUE_P (vi) || sel_deps_has_dep_p[DEPS_IN_INSN]) + if (VINSN_UNIQUE_P (vi)) + /* We can move UNIQUE insn up only as a whole and unchanged, + so it shouldn't have any dependencies. */ return MOVEUP_RHS_NULL; - else - gcc_assert (!VINSN_UNIQUE_P (vi)); } + if (full_ds != 0 && can_overcome_dep_p (full_ds)) + { + if (speculate_expr (insn_to_move_up, full_ds)) + /* Speculation was successful. */ + { + full_ds = 0; + sel_clear_has_dependence (); + } + } + + if (has_dep_p[DEPS_IN_INSN]) + /* We have some dependency that cannot be discarded. */ + return MOVEUP_RHS_NULL; + + if (has_dep_p[DEPS_IN_LHS] && !EXPR_SEPARABLE_P (insn_to_move_up)) + return MOVEUP_RHS_NULL; + /* If dependency is in lhs, it affects only those insns that are not RHS_SCHEDULE_AS_RHS, so they couldn't be moved. Rhses can be moved up through the anti or output dependence @@ -1361,14 +1615,6 @@ moveup_rhs (rhs_t insn_to_move_up, insn_t through_insn) x = z*2; y = z*2; z*2 can be easily lifted above the y=x assignment. */ - if (!RHS_SCHEDULE_AS_RHS (insn_to_move_up) - && (sel_deps_has_dep_p[DEPS_IN_LHS] - || sel_deps_has_dep_p[DEPS_IN_INSN])) - return MOVEUP_RHS_NULL; - - /* rcode 'increases' in the following order: - MOVEUP_RHS_SAME -> MOVEUP_RHS_CHANGED -> MOVEUP_RHS_NULL */ - rcode = MOVEUP_RHS_SAME; /* At this point we have either separable insns, that will be lifted up only as RHSes, or non-separable insns with no dependency in lhs. @@ -1385,13 +1631,23 @@ moveup_rhs (rhs_t insn_to_move_up, insn_t through_insn) In Ex.2 y*2 also can be substituted for x*2, but only the right hand side can be moved because of the output dependency. The operation was cropped to its rhs above. */ - if (sel_deps_has_dep_p[DEPS_IN_RHS]) + if (has_dep_p[DEPS_IN_RHS]) { + ds_t *rhs_dsp = &has_dep_p[DEPS_IN_RHS]; + /* Can't substitute UNIQUE VINSNs. */ gcc_assert (!VINSN_UNIQUE_P (vi)); - if (flag_sel_sched_substitution - && insn_eligible_for_subst_p (through_insn)) + if (can_overcome_dep_p (*rhs_dsp)) + { + if (speculate_expr (insn_to_move_up, *rhs_dsp)) + /* Speculation was successful. */ + *rhs_dsp = 0; + else + return MOVEUP_RHS_NULL; + } + else if (flag_sel_sched_substitution + && insn_eligible_for_subst_p (through_insn)) { /* Substitute in vinsn. */ line_start (); @@ -1401,25 +1657,28 @@ moveup_rhs (rhs_t insn_to_move_up, insn_t through_insn) sel_print_rtl (through_insn); print ("After: "); - gcc_assert (!VINSN_UNIQUE_P (vi)); - if (substitute_rhs (insn_to_move_up, through_insn)) - { - rcode = MOVEUP_RHS_CHANGED; - sel_print_rtl (insn); - } + if (substitute_reg_in_rhs (insn_to_move_up, through_insn)) + sel_print_rtl (insn); else { - rcode = MOVEUP_RHS_NULL; print ("Can't move up due to architecture constraints.\n"); + return MOVEUP_RHS_NULL; } line_finish (); } else - rcode = MOVEUP_RHS_NULL; + return MOVEUP_RHS_NULL; } - return rcode; + /* Don't move trapping insns through jumps. + This check should be at the end to give a chance to control speculation + to perform its duties. */ + if (VINSN_MAY_TRAP_P (EXPR_VINSN (insn_to_move_up)) + && !sel_insn_has_single_succ_p (through_insn, SUCCS_ALL)) + return MOVEUP_RHS_NULL; + + return MOVEUP_RHS_CHANGED; } /* Moves an av set AVP up through INSN, performing necessary @@ -1447,14 +1706,24 @@ moveup_set_rhs (av_set_t *avp, insn_t insn) case MOVEUP_RHS_CHANGED: print (" - changed"); - /* If the resulting insn after substitution is already in av_set, - remove it. */ - if (av_set_lookup_other_equiv_rhs (*avp, rhs)) - { - av_set_iter_remove (&i); - print (" and removed, because av_set already had" \ - " the resulting rhs."); - } + { + rhs_t rhs2 = av_set_lookup_other_equiv_rhs (*avp, RHS_VINSN (rhs)); + + /* If the resulting insn after substitution is already in av_set, + remove it. */ + if (rhs2 != NULL) + { + merge_expr (rhs2, rhs); + + av_set_iter_remove (&i); + print (" and removed."); + + rhs = rhs2; + } + + print (" result: "); + dump_rhs (rhs); + } break; case MOVEUP_RHS_SAME: print (" - unchanged"); @@ -1520,26 +1789,33 @@ equal_after_moveup_path_p_1 (rhs_t rhs, ilist_t path) static bool equal_after_moveup_path_p (rhs_t cur_rhs, ilist_t path, rhs_t rhs_vliw) { - struct _rhs tmp; - bool res; + expr_def _tmp, *tmp = &_tmp; + bool res_p; - rhs_init (&tmp, RHS_VINSN (cur_rhs), 0, RHS_PRIORITY (cur_rhs)); + copy_expr (tmp, cur_rhs); if (path) { ilist_t path1 = ilist_invert (path); - res = equal_after_moveup_path_p_1 (&tmp, path1); + res_p = equal_after_moveup_path_p_1 (tmp, path1); ilist_clear (&path1); } else - res = true; + res_p = true; + + if (res_p) + { + vinsn_t tmp_vinsn = RHS_VINSN (tmp); + vinsn_t rhs_vliw_vinsn = RHS_VINSN (rhs_vliw); - res &= rhs_equal_p (&tmp, rhs_vliw); + if (tmp_vinsn != rhs_vliw_vinsn) + res_p = vinsns_correlate_as_rhses_p (tmp_vinsn, rhs_vliw_vinsn); + } - rhs_clear (&tmp); + clear_expr (tmp); - return res; + return res_p; } @@ -1555,7 +1831,7 @@ equal_after_moveup_path_p (rhs_t cur_rhs, ilist_t path, rhs_t rhs_vliw) In the resulting set we will have only expressions that don't have delay stalls and nonsubstitutable dependences. */ static av_set_t -compute_av_set (insn_t insn, ilist_t p, int ws, int flags) +compute_av_set (insn_t insn, ilist_t p, int ws, bool unique_p) { av_set_t av1; av_set_t rhs_in_all_succ_branches; @@ -1586,7 +1862,7 @@ compute_av_set (insn_t insn, ilist_t p, int ws, int flags) line_finish (); block_finish (); - return flags ? av_set_copy (AV_SET (insn), flags) : AV_SET (insn); + return unique_p ? av_set_copy (AV_SET (insn)) : AV_SET (insn); } /* If the window size exceeds at insn during the first computation of @@ -1634,7 +1910,7 @@ compute_av_set (insn_t insn, ilist_t p, int ws, int flags) av_set_t succ_set; /* We will edit SUCC_SET and RHS_SPEC field of its elements. */ - succ_set = compute_av_set (succs[succ], p, ws + 1, UNIQUE_RHSES); + succ_set = compute_av_set (succs[succ], p, ws + 1, true); if (real_succs_n > 1) { @@ -1645,15 +1921,14 @@ compute_av_set (insn_t insn, ilist_t p, int ws, int flags) into rhs_in_all_succ_branches. This set will be used later for calculating speculation attributes of RHS'es. */ if (succ == 0) - rhs_in_all_succ_branches = av_set_copy (succ_set, - UNIQUE_RHSES); + rhs_in_all_succ_branches = av_set_copy (succ_set); else { av_set_iterator i; rhs_t rhs; FOR_EACH_RHS_1 (rhs, i, &rhs_in_all_succ_branches) - if (!av_set_is_in_p (succ_set, rhs)) + if (!av_set_is_in_p (succ_set, RHS_VINSN (rhs))) av_set_iter_remove (&i); } } @@ -1671,7 +1946,7 @@ compute_av_set (insn_t insn, ilist_t p, int ws, int flags) /* Increase the spec attribute of all RHS'es that didn't come from all successors. */ FOR_EACH_RHS (rhs, i, av1) - if (!av_set_is_in_p (rhs_in_all_succ_branches, rhs)) + if (!av_set_is_in_p (rhs_in_all_succ_branches, RHS_VINSN (rhs))) RHS_SPEC (rhs)++; av_set_clear (&rhs_in_all_succ_branches); @@ -1695,18 +1970,31 @@ compute_av_set (insn_t insn, ilist_t p, int ws, int flags) /* Then, compute av1 above insn. */ if (!INSN_NOP_P (insn)) { + expr_t expr; + vinsn_t vi = INSN_VI (insn); + moveup_set_rhs (&av1, insn); - /* Add this insn to its av set. If a similar insn is already in - av set, we need to change its RHS properly. By now, just - remove it first. */ - av_set_remove_rhs_with_insn (&av1, insn); - av_set_add_insn (&av1, insn); + expr = av_set_lookup (av1, vi); + + if (expr != NULL) + /* ??? It is not clear if we should replace or merge exprs in this + case. But it looks reasonable to me that if we unify (merge) all + operations beyond current point then we should find them all during + move_op () - which is not the case yet. On the other hand if during + move_op () we find only the first operation, then we should replace + the expression here - which is the case. */ + { + clear_expr (expr); + copy_expr (expr, INSN_EXPR (insn)); + } + else + av_set_add (&av1, INSN_EXPR (insn)); } line_start (); print ("insn: "); - dump_insn_1 (insn, DUMP_INSN_UID | DUMP_INSN_PATTERN); + dump_insn_1 (insn, 1); line_finish (); line_start (); @@ -1718,11 +2006,11 @@ compute_av_set (insn_t insn, ilist_t p, int ws, int flags) av_set_clear (&AV_SET (insn)); /* If insn is a bb_header, leave a copy of av1 here. */ - if (bb_header_p (insn)) + if (sel_bb_header_p (insn)) { print ("Save av(%d) in bb header", INSN_UID (insn)); - AV_SET (insn) = flags ? av_set_copy (av1, UNIQUE_RHSES) : av1; + AV_SET (insn) = unique_p ? av_set_copy (av1) : av1; AV_LEVEL (insn) = global_level; } @@ -1740,7 +2028,7 @@ propagate_lv_set (regset lv, insn_t insn) return; /* LV1 = LV1 \ { DEST (insn) } */ - if (!INSN_P (insn) || GET_CODE (PATTERN (insn)) != COND_EXEC) + if (GET_CODE (PATTERN (insn)) != COND_EXEC) /* May-defs should not kill other sets. */ AND_COMPL_REG_SET (lv, INSN_REG_SETS (insn)); @@ -1802,7 +2090,7 @@ compute_live (insn_t insn) /* Compute live set above INSN. */ propagate_lv_set (lv, insn); - if (bb_header_p (insn)) + if (sel_bb_header_p (insn)) { gcc_assert (LV_SET (insn) != NULL); @@ -1837,7 +2125,7 @@ update_data_sets (rtx insn) { gcc_assert (LV_SET (insn) != NULL && INSN_AV_VALID_P (insn) - && bb_header_p (insn)); + && sel_bb_header_p (insn)); /* Recompute the first LV_SET as it may have got invalid. */ ignore_first = true; @@ -1862,13 +2150,38 @@ update_data_sets (rtx insn) /* Invalidate AV_SET. */ AV_LEVEL (insn) = 0; - if (bb_header_p (insn)) + if (sel_bb_header_p (insn)) compute_av_set (insn, NULL, 0, 0); /* If INSN is not a bb_header any longer, its av_set will be deleted on the next compute_av_set (). */ } +/* Helper for move_op () and find_used_regs (). + Return speculation type for which a check should be created on the place + of INSN. EXPR is one of the original ops we are searching for. */ +static ds_t +get_spec_check_type_for_insn (insn_t insn, expr_t expr) +{ + ds_t to_check_ds; + ds_t already_checked_ds = EXPR_SPEC_DONE_DS (INSN_EXPR (insn)); + + to_check_ds = EXPR_SPEC_TO_CHECK_DS (expr); + + if (targetm.sched.get_insn_checked_ds) + already_checked_ds |= targetm.sched.get_insn_checked_ds (insn); + + if (spec_info != NULL + && (spec_info->flags & SEL_SCHED_SPEC_DONT_CHECK_CONTROL)) + already_checked_ds |= BEGIN_CONTROL; + + already_checked_ds = ds_get_speculation_types (already_checked_ds); + + to_check_ds &= ~already_checked_ds; + + return to_check_ds; +} + /* Functions to check liveness restrictions on available instructions. */ /* Helper function for find_used_regs. @@ -1939,7 +2252,7 @@ find_used_regs_1 (insn_t insn, av_set_t orig_ops, ilist_t path, return 0; } - orig_ops = av_set_copy (orig_ops, UNIQUE_RHSES); + orig_ops = av_set_copy (orig_ops); /* If we've found valid av set, then filter the orig_ops set. */ if (INSN_AV_VALID_P (insn)) { @@ -1960,7 +2273,9 @@ find_used_regs_1 (insn_t insn, av_set_t orig_ops, ilist_t path, return 0; } - av_set_leave_one (&orig_ops); + /* !!! When using different types of speculation we must not leave + just one element in orig_ops. */ + /*av_set_leave_one (&orig_ops);*/ } if (CALL_P (insn)) @@ -1979,7 +2294,7 @@ find_used_regs_1 (insn_t insn, av_set_t orig_ops, ilist_t path, When traversing the DAG below this insn is finished, insert bookkeeping code, if the insn is a joint point, and remove leftovers. */ - rhs = av_set_lookup_insn (orig_ops, insn); + rhs = av_set_lookup (orig_ops, INSN_VI (insn)); if (rhs) { /* We have found the original operation. Mark the registers that do not @@ -1990,7 +2305,13 @@ find_used_regs_1 (insn_t insn, av_set_t orig_ops, ilist_t path, print ("found original operation!"); - def_list_add (original_insns, insn, crosses_call); + { + bool needs_spec_check_p; + + needs_spec_check_p = (get_spec_check_type_for_insn (insn, rhs) != 0); + + def_list_add (original_insns, insn, crosses_call, needs_spec_check_p); + } res = 1; is_orig_op = true; @@ -2016,7 +2337,7 @@ find_used_regs_1 (insn_t insn, av_set_t orig_ops, ilist_t path, REG_DEAD: dx */ /* FIXME: see comment above and enable MEM_P in vinsn_separable_p. */ - gcc_assert (!VINSN_SEPARABLE (INSN_VI (insn)) + gcc_assert (!VINSN_SEPARABLE_P (INSN_VI (insn)) || !MEM_P (INSN_LHS (insn))); } else @@ -2026,28 +2347,29 @@ find_used_regs_1 (insn_t insn, av_set_t orig_ops, ilist_t path, bool mutexed; rhs_t r; av_set_iterator avi; - int succs_num; + regset live_on_other_branches = NULL; res = 0; - if (flag_sel_sched_substitution) { - /* Av set ops could have changed when moving through this insn. - To find them below it, we have to un-substitute them. */ - un_substitute (&orig_ops, insn); - - /* If all original opernads have been filtered on this branch, - return. */ - if (!orig_ops) - { - block_finish (); - return 0; - } - } + /* Av set ops could have changed when moving through this insn. + To find them below it, we have to un-speculate and un-substitute + them. */ + + un_speculate (&orig_ops, insn); + + un_substitute (&orig_ops, insn); + + /* If all original operands have been filtered on this branch, + return. */ + if (!orig_ops) + { + block_finish (); + return 0; + } /* Continue searching. Do recursion here. */ ilist_add (&path, insn); - succs_num = cfg_succs_n (insn, SUCCS_NORMAL); FOR_EACH_SUCC (succ, succ_i, insn) { int b; @@ -2065,23 +2387,38 @@ find_used_regs_1 (insn_t insn, av_set_t orig_ops, ilist_t path, original_insns); if (b == 0) + /* Mark all registers that do not meet the following condition: + (3) not live on the other path of any conditional branch + that is passed by the operation, in case original + operations are not present on both paths of the + conditional branch. + + if b is false, then it's not present on that branch. */ { - /* Mark all registers that do not meet the following condition: - (3) not live on the other path of any conditional branch - that is passed by the operation, in case original - operations are not present on both paths of the - conditional branch. - - if b is false, then it's not present on that branch. */ - IOR_REG_SET (used_regs, compute_live (succ)); - } - else if (b == 1) - res = 1; - else - { - gcc_assert (b == -1); - res = -1; + regset succ_live = compute_live (succ); + + if (live_on_other_branches == NULL) + /* We're getting succ_live out of the pool. */ + { + regset tmp = get_regset_from_pool (); + + gcc_assert (tmp == succ_live); + + live_on_other_branches = succ_live; + } + else + /* We're leaving succ_live in the pool. */ + IOR_REG_SET (live_on_other_branches, succ_live); } + else if (res != -1) + res = b; + } + + if (live_on_other_branches != NULL) + { + IOR_REG_SET (used_regs, live_on_other_branches); + return_regset_to_pool (live_on_other_branches); + live_on_other_branches = NULL; } if (res == 0) @@ -2142,9 +2479,14 @@ find_used_regs_1 (insn_t insn, av_set_t orig_ops, ilist_t path, av_set_clear (&orig_ops); - gcc_assert (!bb_header_p (insn) || INSN_AV_VALID_P (insn) + gcc_assert (!sel_bb_header_p (insn) || INSN_AV_VALID_P (insn) || AV_LEVEL (insn) == -1); + if (res == -1 && AV_LEVEL (insn) == -1) + /* INSN has an incorrect av_set on which we've couldn't have filtered + ORIG_OPS. */ + res = 0; + block_finish (); return res; } @@ -2170,7 +2512,6 @@ find_used_regs (insn_t insn, av_set_t orig_ops, regset used_regs, def_t def; int res; - res = find_used_regs_1 (insn, orig_ops, NULL, used_regs, unavailable_hard_regs, false, original_insns); @@ -2187,8 +2528,14 @@ find_used_regs (insn_t insn, av_set_t orig_ops, regset used_regs, unavailable_hard_regs. */ FOR_EACH_DEF (def, i, *original_insns) { - if (VINSN_SEPARABLE (INSN_VI (def->orig_insn))) + vinsn_t vinsn = INSN_VI (def->orig_insn); + + if (VINSN_SEPARABLE_P (vinsn)) mark_unavailable_hard_regs (def, unavailable_hard_regs, used_regs); + + if (def->needs_spec_check_p) + /* Do not allow clobbering of ld.[sa] address. */ + IOR_REG_SET (used_regs, VINSN_REG_USES (vinsn)); } return true; @@ -2225,25 +2572,48 @@ sel_rank_for_schedule (const void *x, const void *y) } /* Prefer not scheduled insn over scheduled one. */ - val = (VINSN_SCHED_TIMES (tmp_vinsn) - - VINSN_SCHED_TIMES (tmp2_vinsn)); + val = EXPR_SCHED_TIMES (tmp) - EXPR_SCHED_TIMES (tmp2); if (val) return val; /* FIXME: Want to try first clear all insns except with minimal SPEC. */ /* Prefer a rhs with smaller SPEC attribute. FIXME: should we use probabilities here? */ - val = RHS_SPEC (tmp) - RHS_SPEC (tmp2); + val = EXPR_SPEC (tmp) - EXPR_SPEC (tmp2); if (val) return val; - /* Prefer a rhs with greater priority. */ - val = RHS_PRIORITY (tmp2) - RHS_PRIORITY (tmp); + /* Prefer an expr with greater priority. */ + val = EXPR_PRIORITY (tmp2) - EXPR_PRIORITY (tmp); if (val) return val; - tmp_insn = RHS_INSN (tmp); - tmp2_insn = RHS_INSN (tmp2); + if (spec_info != NULL && spec_info->mask != 0) + /* This code was taken from haifa-sched.c: rank_for_schedule (). */ + { + ds_t ds1, ds2; + dw_t dw1, dw2; + int dw; + + ds1 = EXPR_SPEC_DONE_DS (tmp); + if (ds1) + dw1 = ds_weak (ds1); + else + dw1 = NO_DEP_WEAK; + + ds2 = EXPR_SPEC_DONE_DS (tmp2); + if (ds2) + dw2 = ds_weak (ds2); + else + dw2 = NO_DEP_WEAK; + + dw = dw2 - dw1; + if (dw > (NO_DEP_WEAK / 8) || dw < -(NO_DEP_WEAK / 8)) + return dw; + } + + tmp_insn = EXPR_INSN_RTX (tmp); + tmp2_insn = EXPR_INSN_RTX (tmp2); /* Prefer an old insn to a bookkeeping insn. */ if (INSN_UID (tmp_insn) < first_emitted_uid @@ -2270,12 +2640,8 @@ fill_ready_list (av_set_t av, bnd_t bnd, fence_t fence) av_set_iterator si; int n, i, stalled, sched_next_worked = 0; deps_t dc = BND_DC (bnd); - - /* Allocate the vector for sorting the av set. */ - if (!vec_av_set) - vec_av_set = VEC_alloc (rhs_t, heap, 5); - else - gcc_assert (VEC_empty (rhs_t, vec_av_set)); + bool try_data_p = true; + bool try_control_p = true; /* Fill the vector with recognizable insns. */ FOR_EACH_RHS (rhs, si, av) @@ -2287,16 +2653,55 @@ fill_ready_list (av_set_t av, bnd_t bnd, fence_t fence) { /* Do not pipeline these insns when they were already scheduled; as we emit them unconditionally, it leads to an infinite loop. */ - if (VINSN_SCHED_TIMES (RHS_VINSN (rhs)) > 0) - { - gcc_assert (pipelining_p); - continue; - } - - VEC_block_remove (rhs_t, vec_av_set, 0, - VEC_length (rhs_t, vec_av_set)); - return rhs; + if (EXPR_SCHED_TIMES (rhs) <= 0) + return rhs; + else + gcc_assert (pipelining_p); } + else + { + if (spec_info != NULL) + { + ds_t ds = EXPR_SPEC_DONE_DS (rhs); + + if ((spec_info->flags & PREFER_NON_DATA_SPEC) + && !(ds & BEGIN_DATA)) + try_data_p = false; + + if ((spec_info->flags & PREFER_NON_CONTROL_SPEC) + && !(ds & BEGIN_CONTROL)) + try_control_p = false; + } + } + } + + /* Allocate the vector for sorting the av set. */ + if (!vec_av_set) + vec_av_set = VEC_alloc (rhs_t, heap, 5); + else + gcc_assert (VEC_empty (rhs_t, vec_av_set)); + + FOR_EACH_RHS (rhs, si, av) + { + ds_t ds = EXPR_SPEC_DONE_DS (rhs); + + if (ds & SPECULATIVE) + { + if ((ds & BEGIN_DATA) && !try_data_p) + /* We don't want any data speculative instructions right now. */ + continue; + + if ((ds & BEGIN_CONTROL) && !try_control_p) + /* We don't want any control speculative instructions right + now. */ + continue; + } + + if (EXPR_SCHED_TIMES (rhs) + >= PARAM_VALUE (PARAM_SELSCHED_MAX_SCHED_TIMES)) + /* Don't pipeline already pipelined code as that would increase + number of unnecessary register moves. */ + continue; VEC_safe_push (rhs_t, heap, vec_av_set, rhs); } @@ -2321,10 +2726,10 @@ fill_ready_list (av_set_t av, bnd_t bnd, fence_t fence) ready.vec = xrealloc (ready.vec, ready.veclen * sizeof (*ready.vec)); } - if (ready.veclen > old_ready_veclen) + if (ready.veclen > max_issue_size) { - sched_local_init (ready.veclen); - old_ready_veclen = ready.veclen; + max_issue_size = ready.veclen; + sched_extend_ready_list (ready.veclen); } line_start (); @@ -2332,18 +2737,26 @@ fill_ready_list (av_set_t av, bnd_t bnd, fence_t fence) for (i = 0, n = 0, stalled = 0; VEC_iterate (rhs_t, vec_av_set, i, rhs); i++) { - vinsn_t vi = RHS_VINSN (rhs); - insn_t insn = VINSN_INSN (vi); + vinsn_t vi = EXPR_VINSN (rhs); + insn_t insn = VINSN_INSN_RTX (vi); + + dump_rhs (rhs); /* Don't allow insns from a SCHED_GROUP to be scheduled if their - ancestors havn't been scheduled. */ - if (VINSN_UNIQUE_P (vi) && SCHED_GROUP_P (insn) && !bb_header_p (insn)) + ancestors havn't been scheduled. + !!! This should be dealt with in moveup_rhs (). */ + if (VINSN_UNIQUE_P (vi) && SCHED_GROUP_P (insn) + && !sel_bb_header_p (insn)) { insn_t prev = PREV_INSN (insn); if (SCHED_GROUP_P (prev) && INSN_SCHED_CYCLE (prev) <= INSN_SCHED_CYCLE (insn)) - continue; + { + /* Dealt in moveup_rhs (). */ + gcc_unreachable (); + continue; + } } /* Don't allow any insns other than from SCHED_GROUP if we have one. */ @@ -2356,7 +2769,7 @@ fill_ready_list (av_set_t av, bnd_t bnd, fence_t fence) sched_next_worked++; /* Don't allow any insns whose data is not yet ready. */ - if (!tick_check_p (vi, dc, fence)) + if (!tick_check_p (rhs, dc, fence)) { /* We need to know whether we do need to stall for any insns. */ stalled++; @@ -2366,8 +2779,12 @@ fill_ready_list (av_set_t av, bnd_t bnd, fence_t fence) ready_try[n] = 0; ready.vec[n] = insn; n++; + + GET_VINSN_BY_INSN (insn) = RHS_VINSN (rhs); } + line_finish (); + ready.n_ready = n; ready.first = n - 1; @@ -2393,8 +2810,6 @@ fill_ready_list (av_set_t av, bnd_t bnd, fence_t fence) FENCE_SCHED_NEXT (fence) = NULL_RTX; } - line_finish (); - /* Clear the vector for future use. */ VEC_block_remove (rhs_t, vec_av_set, 0, VEC_length (rhs_t, vec_av_set)); @@ -2430,59 +2845,42 @@ sel_dfa_new_cycle (insn_t insn, fence_t fence) } /* Choose the best rhs (BEST_RHS_VLIW) from *AV_VLIW_PTR and a suitable - register for it (BEST_REG_FOUND). BNDS and FENCE are current boundaries + register for it (BEST_REGNOP). BNDS and FENCE are current boundaries and scheduling fence respectively. */ static void -find_best_rhs_and_reg_that_fits (av_set_t *av_vliw_ptr, blist_t bnds, - fence_t fence, rhs_t *best_rhs_vliw, - rtx *best_reg_found) +find_best_rhs_and_reg_that_fits (av_set_t *av_vliw_ptr, blist_t bnds, + fence_t fence, rhs_t *best_rhs_vliw) { rhs_t res = NULL; insn_t best = NULL_RTX; - rtx best_reg; rhs_t r; - _list_iterator i3; + av_set_iterator av_it; /* First we need to choose destination registers for each rhs. Multiple rhses can be given the same register, because the choice is local and there's only one register from av_vliw that will be scheduled on this cycle. */ - FOR_EACH_RHS_1 (r, i3, av_vliw_ptr) + FOR_EACH_RHS_1 (r, av_it, av_vliw_ptr) { - /* BEST_REG will be -1 if: - - RHS_SCHEDULE_AS_RHS is true but we were unable to find suitable - register; - - RHS_SCHEDULE_AS_RHS is false but the insn sets/clobbers one of - the registers that are used on the moving path. */ - best_reg = find_best_reg_for_rhs (r, bnds); - - /* If we've found suitable register for replacement, and - RHS_SCHEDULE_AS_RHS is true, replace the reg and see if the - resulting insn is valid. */ - - if (best_reg) - { - if (RHS_SCHEDULE_AS_RHS (r)) - { + { + ds_t ds = EXPR_SPEC_DONE_DS (r); - /* After reload we must choose only hard registers. */ - gcc_assert (!reload_completed || HARD_REGISTER_P (best_reg)); + if ((ds & SPECULATIVE) + && ds_weak (ds) < spec_info->weakness_cutoff) + /* The probability of a success is too low - don't speculate. */ + { + av_set_iter_remove (&av_it); + continue; + } + } - /* If the register we have chosen is different, replace it. */ - if (rhs_dest_regno (r) != REGNO (best_reg)) - replace_dest_with_reg_in_rhs (r, best_reg); - - /* The resulting insn is valid because after BEST_REG was - chosen, insn_valid_p has been called. */ - gcc_assert (insn_valid_p (RHS_INSN (r))); - } - } - else + if (!find_best_reg_for_rhs (r, bnds)) + /* If can't choose a register for the rhs (liveness or + architectural restrictions for for the insn don't allow that), + expr should be removed from AV_VLIW. */ { - /* If can't choose register for the rhs (liveness or - architectural restrictions for for the insn are violated), - best_reg is -1 and the rhs is removed here. */ - av_set_iter_remove (&i3); + av_set_iter_remove (&av_it); + continue; } } @@ -2558,13 +2956,12 @@ find_best_rhs_and_reg_that_fits (av_set_t *av_vliw_ptr, blist_t bnds, if (can_issue_more == 0) { *best_rhs_vliw = NULL; - *best_reg_found = NULL_RTX; return; } min_spec_insn = ready_element (&ready, 0); - min_spec_vinsn = INSN_VI (min_spec_insn); - min_spec_rhs = av_set_lookup_insn (*av_vliw_ptr, min_spec_insn); + min_spec_vinsn = GET_VINSN_BY_INSN (min_spec_insn); + min_spec_rhs = av_set_lookup (*av_vliw_ptr, min_spec_vinsn); gcc_assert (min_spec_rhs != NULL && RHS_VINSN (min_spec_rhs) == min_spec_vinsn); @@ -2572,7 +2969,8 @@ find_best_rhs_and_reg_that_fits (av_set_t *av_vliw_ptr, blist_t bnds, print ("ready after reorder: "); ready_try [0] = 0; - dump_insn (min_spec_insn); + + dump_vinsn (min_spec_vinsn); print (":0; "); if (targetm.sched.first_cycle_multipass_dfa_lookahead_guard) @@ -2589,9 +2987,10 @@ find_best_rhs_and_reg_that_fits (av_set_t *av_vliw_ptr, blist_t bnds, gcc_assert (ready_try [i] == 0); ready_try[i] = r; - dump_insn (insn); + dump_vinsn (GET_VINSN_BY_INSN (insn)); print (":%d; ", ready_try[i]); } + line_finish (); privileged_n = 0; @@ -2609,19 +3008,15 @@ find_best_rhs_and_reg_that_fits (av_set_t *av_vliw_ptr, blist_t bnds, cur_insn = ready_element (&ready, i); - cur_vinsn = INSN_VI (cur_insn); + cur_vinsn = GET_VINSN_BY_INSN (cur_insn); - cur_rhs = av_set_lookup_insn (*av_vliw_ptr, cur_insn); + cur_rhs = av_set_lookup (*av_vliw_ptr, cur_vinsn); gcc_assert (cur_rhs != NULL && RHS_VINSN (cur_rhs) == cur_vinsn); if (RHS_SPEC (cur_rhs) > RHS_SPEC (min_spec_rhs)) break; - if (VINSN_SCHED_CYCLE (cur_vinsn) - > VINSN_SCHED_CYCLE (min_spec_vinsn)) - break; - ++privileged_n; } @@ -2645,8 +3040,9 @@ find_best_rhs_and_reg_that_fits (av_set_t *av_vliw_ptr, blist_t bnds, { /* We can't use max_issue; just return the first element. */ insn_t insn = ready_element (&ready, 0); + vinsn_t vinsn = GET_VINSN_BY_INSN (insn); - if (dfa_cost (insn, fence) >= 1) + if (vinsn_dfa_cost (vinsn, fence) >= 1) { can_issue = 0; index = -1; @@ -2662,7 +3058,7 @@ find_best_rhs_and_reg_that_fits (av_set_t *av_vliw_ptr, blist_t bnds, if (can_issue) { best = ready_element (&ready, index); - res = av_set_lookup_insn (*av_vliw_ptr, best); + res = av_set_lookup (*av_vliw_ptr, GET_VINSN_BY_INSN (best)); gcc_assert (res); } else @@ -2697,15 +3093,54 @@ find_best_rhs_and_reg_that_fits (av_set_t *av_vliw_ptr, blist_t bnds, *best_rhs_vliw = res; - if (res && RHS_SCHEDULE_AS_RHS (res)) - *best_reg_found = RHS_DEST (res); - else - *best_reg_found = NULL_RTX; - return; } +/* Emit an instruction from EXPR with SEQNO after PLACE_TO_INSERT. */ +static insn_t +gen_insn_from_expr_after (expr_t expr, int seqno, insn_t place_to_insert) +{ + { + insn_t insn = RHS_INSN (expr); + + /* This assert fails when we have identical instructions + one of which dominates the other. In this case move_op () + finds the first instruction and doesn't search for second one. + The solution would be to compute av_set after the first found + insn and, if insn present in that set, continue searching. + For now we workaround this issue in move_op. */ + gcc_assert (!INSN_IN_STREAM_P (insn)); + + gcc_assert (!LV_SET_VALID_P (insn)); + } + + { + rtx reg = expr_dest_reg (expr); + + if (reg != NULL_RTX) + { + static int reg_rename_this_tick = 0; + + if (HARD_REGISTER_P (reg)) + { + unsigned regno = REGNO (reg); + + reg_rename_tick[regno] = ++reg_rename_this_tick; + regs_ever_live[regno] = 1; + } + } + } + + { + insn_t new_insn; + + new_insn = sel_gen_insn_from_expr_after (expr, seqno, place_to_insert); + + return new_insn; + } +} + /* Functions that implement the core of the scheduler. */ /* Generate a bookkeeping copy of "REG = CUR_RHS" insn at JOIN_POINT on the @@ -2718,20 +3153,14 @@ find_best_rhs_and_reg_that_fits (av_set_t *av_vliw_ptr, blist_t bnds, newly created bb, which is the lower bb. All scheduler data is initialized for the newly created insn. */ static basic_block -generate_bookkeeping_insn (rhs_t c_rhs, rtx reg, insn_t join_point, edge e1, - edge e2) +generate_bookkeeping_insn (rhs_t c_rhs, insn_t join_point, edge e1, edge e2) { basic_block src, bb = e2->dest; - basic_block new_bb, res = NULL, empty_bb = e1->dest; - vinsn_t book_insn = RHS_VINSN (c_rhs); - insn_t book_insn2; - insn_t new_insn, place_to_insert, src_end; - - if (reg && RHS_SCHEDULE_AS_RHS (c_rhs) - && rhs_dest_regno (c_rhs) != REGNO (reg)) - book_insn2 = replace_dest_with_reg_in_vinsn (book_insn, reg, true); - else - book_insn2 = expr_copy (VINSN_PATTERN (book_insn)); + basic_block new_bb, res = NULL; + insn_t src_end = NULL_RTX; + insn_t place_to_insert; + /* Save the original destination of E1. */ + basic_block empty_bb = e1->dest; print ("generate_bookkeeping_insn(%d->%d)", e1->src->index, e2->dest->index); @@ -2747,13 +3176,18 @@ generate_bookkeeping_insn (rhs_t c_rhs, rtx reg, insn_t join_point, edge e1, predecessor block of BB. */ src = EDGE_PRED (bb, 0) == e2 ? EDGE_PRED (bb, 1)->src : EDGE_PRED (bb, 0)->src; + /* Instruction, after which we would try to insert bookkeeping insn. */ src_end = BB_END (src); gcc_assert (in_current_region_p (src)); - /* Cannot create bookkeeping after jumps. */ - if (INSN_P (src_end) && control_flow_insn_p (src_end)) - src = NULL; + if (INSN_P (src_end)) + { + if (control_flow_insn_p (src_end)) + src = NULL; + } + else + gcc_assert (NOTE_INSN_BASIC_BLOCK_P (src_end)); } else src = NULL; @@ -2792,7 +3226,6 @@ generate_bookkeeping_insn (rhs_t c_rhs, rtx reg, insn_t join_point, edge e1, can_add_real_insns_p = true; insn_init.what = INSN_INIT_WHAT_INSN; - insn_init.todo = INSN_INIT_TODO_ALL | INSN_INIT_TODO_LV_SET; /* Make a jump skipping bookkeeping copy. */ if (e1->flags & EDGE_FALLTHRU) @@ -2801,7 +3234,8 @@ generate_bookkeeping_insn (rhs_t c_rhs, rtx reg, insn_t join_point, edge e1, sel_redirect_edge_and_branch (e1, new_bb); gcc_assert (e1->dest == new_bb); - gcc_assert (bb_empty_p (bb)); + + gcc_assert (sel_bb_empty_p (bb)); place_to_insert = BB_END (bb); } @@ -2812,33 +3246,50 @@ generate_bookkeeping_insn (rhs_t c_rhs, rtx reg, insn_t join_point, edge e1, while (EDGE_COUNT (empty_bb->preds) == 0) { basic_block next_bb = empty_bb->next_bb; - sel_remove_empty_bb (empty_bb, false); + sel_remove_empty_bb (empty_bb, false, true); empty_bb = next_bb; } - insn_init.what = INSN_INIT_WHAT_INSN; - insn_init.todo = INSN_INIT_TODO_ALL & ~INSN_INIT_TODO_SEQNO; - new_insn = emit_insn_after (book_insn2, place_to_insert); - gcc_assert ((!src && BB_END (bb) == new_insn && bb_header_p (new_insn)) - || BB_END (src) == new_insn); + { + rtx new_insn_rtx; + vinsn_t new_vinsn; + expr_def _new_expr, *new_expr = &_new_expr; + insn_t new_insn; - gcc_assert (AV_SET (new_insn) == NULL && AV_LEVEL (new_insn) == 0); + new_insn_rtx = create_copy_of_insn_rtx (EXPR_INSN_RTX (c_rhs)); + new_vinsn = create_vinsn_from_insn_rtx (new_insn_rtx); + copy_expr (new_expr, c_rhs); + change_vinsn_in_expr (new_expr, new_vinsn); - INSN_PRIORITY (new_insn) = RHS_PRIORITY (c_rhs); - INSN_COST (new_insn) = insn_cost (new_insn, 0, 0); - INSN_SEQNO (new_insn) = INSN_SEQNO (join_point); + if (EXPR_SCHED_TIMES (new_expr) + >= PARAM_VALUE (PARAM_SELSCHED_MAX_SCHED_TIMES)) + /* To make scheduling of this bookkeeping copy possible we decrease + its scheduling counter. */ + --EXPR_SCHED_TIMES (new_expr); - gcc_assert (LV_SET (join_point) != NULL); + new_insn = gen_insn_from_expr_after (new_expr, INSN_SEQNO (join_point), + place_to_insert); - if (bb_header_p (new_insn)) - { - LV_SET (new_insn) = get_regset_from_pool (); - ignore_first = true; - compute_live (new_insn); - } + clear_expr (new_expr); + + gcc_assert ((src == NULL && BB_END (bb) == new_insn + && sel_bb_header_p (new_insn)) + || BB_END (src) == new_insn); + + gcc_assert (AV_SET (new_insn) == NULL && AV_LEVEL (new_insn) == 0); - /* Set AV_LEVEL to special value to bypass assert in move_op (). */ - AV_LEVEL (new_insn) = -1; + /* Set AV_LEVEL to special value to bypass assert in move_op (). */ + AV_LEVEL (new_insn) = -1; + + gcc_assert (LV_SET (join_point) != NULL); + + if (sel_bb_header_p (new_insn)) + { + LV_SET (new_insn) = get_regset_from_pool (); + ignore_first = true; + compute_live (new_insn); + } + } return res; } @@ -2855,7 +3306,6 @@ fill_insns (fence_t fence, int seqno, ilist_t **scheduled_insns_tailpp) av_set_t av_vliw = NULL; insn_t insn = FENCE_INSN (fence); state_t temp_state = alloca (dfa_state_size); - rtx best_reg = NULL_RTX; blist_add (&bnds, insn, NULL, FENCE_DC (fence)); bnds_tailp = &BLIST_NEXT (bnds); @@ -2887,15 +3337,15 @@ fill_insns (fence_t fence, int seqno, ilist_t **scheduled_insns_tailpp) av_set_t av1_copy; av_set_clear (&BND_AV (bnd)); - BND_AV (bnd) = compute_av_set (BND_TO (bnd), NULL, 0, UNIQUE_RHSES); + BND_AV (bnd) = compute_av_set (BND_TO (bnd), NULL, 0, true); av_set_clear (&BND_AV1 (bnd)); - BND_AV1 (bnd) = av_set_copy (BND_AV (bnd), UNIQUE_RHSES); + BND_AV1 (bnd) = av_set_copy (BND_AV (bnd)); if (enable_moveup_set_path_p) moveup_set_path (&BND_AV1 (bnd), BND_PTR (bnd)); - av1_copy = av_set_copy (BND_AV1 (bnd), UNIQUE_RHSES); + av1_copy = av_set_copy (BND_AV1 (bnd)); av_set_union_and_clear (&av_vliw, &av1_copy); } while ((bnds1 = BLIST_NEXT (bnds1))); @@ -2903,37 +3353,37 @@ fill_insns (fence_t fence, int seqno, ilist_t **scheduled_insns_tailpp) sel_dump_cfg ("after-compute_av"); /* If debug parameters tell us to ignore this attempt to move an insn, - obey. */ + obey. */ { - int now; - int start; - int stop; - bool do_p; - - now = ++fill_insns_run; - start = PARAM_VALUE (PARAM_INSN_START); - stop = PARAM_VALUE (PARAM_INSN_STOP); - do_p = (PARAM_VALUE (PARAM_INSN_P) == 1); - - if (do_p) - do_p = (start <= now) && (now <= stop); - else - do_p = (start > now) || (now > stop); - - if (!do_p) - /* Leave only the next insn in av_vliw. */ - { - av_set_iterator av_it; - rhs_t expr; - bnd_t bnd = BLIST_BND (bnds); - insn_t next = BND_TO (bnd); + int now; + int start; + int stop; + bool do_p; + + now = ++fill_insns_run; + start = PARAM_VALUE (PARAM_INSN_START); + stop = PARAM_VALUE (PARAM_INSN_STOP); + do_p = (PARAM_VALUE (PARAM_INSN_P) == 1); + + if (do_p) + do_p = (start <= now) && (now <= stop); + else + do_p = (start > now) || (now > stop); + + if (!do_p) + /* Leave only the next insn in av_vliw. */ + { + av_set_iterator av_it; + rhs_t expr; + bnd_t bnd = BLIST_BND (bnds); + insn_t next = BND_TO (bnd); - gcc_assert (BLIST_NEXT (bnds) == NULL); + gcc_assert (BLIST_NEXT (bnds) == NULL); - FOR_EACH_RHS_1 (expr, av_it, &av_vliw) - if (RHS_INSN (expr) != next) - av_set_iter_remove (&av_it); - } + FOR_EACH_RHS_1 (expr, av_it, &av_vliw) + if (RHS_INSN (expr) != next) + av_set_iter_remove (&av_it); + } } /* Now we've computed AV_VLIW - the set of expressions that can be @@ -2949,8 +3399,7 @@ fill_insns (fence_t fence, int seqno, ilist_t **scheduled_insns_tailpp) /* Choose the best expression and, if needed, destination register for it. */ - find_best_rhs_and_reg_that_fits (&av_vliw, bnds, fence, - &rhs_vliw, &best_reg); + find_best_rhs_and_reg_that_fits (&av_vliw, bnds, fence, &rhs_vliw); if (!rhs_vliw) { @@ -2975,6 +3424,7 @@ fill_insns (fence_t fence, int seqno, ilist_t **scheduled_insns_tailpp) bndsp = &bnds; bnds_tailp1 = bnds_tailp; do + /* !!! This code is guaranteed to execute only once. */ { bnd_t bnd = BLIST_BND (*bndsp); av_set_t rhs_seq = NULL; @@ -2982,198 +3432,183 @@ fill_insns (fence_t fence, int seqno, ilist_t **scheduled_insns_tailpp) av_set_iterator i; insn_t *succs; int succs_n; - bool asm_p = false; + bool asm_p; + bool first_p = true; - if (!av_set_is_in_p (BND_AV1 (bnd), rhs_vliw)) + if (!av_set_is_in_p (BND_AV1 (bnd), RHS_VINSN (rhs_vliw))) { bndsp = &BLIST_NEXT (*bndsp); bnds_tail1 = *bnds_tailp1; continue; } - /* !!! This code is guaranteed to execute only once. - When pattern scheduling will be implemented, move - some stuff level up from here (e.g.: state_transition ()). */ FOR_EACH_RHS (rhs, i, BND_AV (bnd)) { ilist_t root = enable_moveup_set_path_p ? BND_PTR (bnd) : NULL; if (equal_after_moveup_path_p (rhs, root, rhs_vliw)) - { - /* RHS_SCHEDULE_AS_RHS attr is copied from the rhs_vliw, - because we decide it upon BND_AV1 (bnd), which goes - into av_vliw. */ - av_set_add_vinsn (&rhs_seq, RHS_VINSN (rhs), RHS_SPEC (rhs), - RHS_PRIORITY (rhs)); - } + { + gcc_assert (first_p); + first_p = false; + /*av_set_add (&rhs_seq, rhs);*/ + } } - gcc_assert (rhs_seq); + av_set_add (&rhs_seq, rhs_vliw); + line_start (); print ("rhs_seq: "); dump_av_set (rhs_seq); line_finish (); - if (vinsn_cond_branch_p (RHS_VINSN (av_set_element (rhs_seq, 0)))) - { - insn = BND_TO (bnd); - } - else - { - insn_t place_to_insert, new_bb_header = NULL; - struct _rhs _c_rhs, *c_rhs = &_c_rhs; - bool b; - - /* Init place_to_insert before calling move_op, as the later - can possibly remove BND_TO (bnd). */ - if (/* If this is not the first insn scheduled. */ - BND_PTR (bnd)) + /* Move choosen insn. */ + { + insn_t place_to_insert; + insn_t new_bb_head = NULL_RTX; + expr_def _c_rhs, *c_rhs = &_c_rhs; + bool b; + + /* Init place_to_insert before calling move_op, as the later + can possibly remove BND_TO (bnd). */ + if (/* If this is not the first insn scheduled. */ + BND_PTR (bnd)) + { + gcc_unreachable (); + /* Add it after last scheduled. */ place_to_insert = ILIST_INSN (BND_PTR (bnd)); - else - /* Add it before BND_TO. The difference is in the - basic block, where INSN will be added. */ - place_to_insert = PREV_INSN (BND_TO (bnd)); - - gcc_assert (!best_reg || GET_MODE (best_reg) != VOIDmode); - - sel_dump_cfg ("before-move_op"); - - /* Marker is useful bind .dot dumps to the log. */ - print_marker_to_log (); + } + else + /* Add it before BND_TO. The difference is in the + basic block, where INSN will be added. */ + place_to_insert = PREV_INSN (BND_TO (bnd)); + + sel_dump_cfg ("before-move_op"); + + /* Marker is useful to bind .dot dumps and the log. */ + print_marker_to_log (); + + /* Make a move. This call will remove the original operation, + insert all necessary bookkeeping instructions and update the + data sets. After that all we have to do is add the operation + at before BND_TO (BND). */ + b = move_op (BND_TO (bnd), rhs_seq, NULL, NULL, NULL, c_rhs); + + /* We should be able to find the expression we've chosen for + scheduling. */ + gcc_assert (b); + + /* Find a place for C_RHS to schedule. + We want to have an invariant that only insns that are + sel_bb_header_p () have a valid LV_SET. But, in the same time, + we don't want overhead from recomputation of compute_live () + for the half of a block after each movement. Resolution of + this is floating bb header that will advance along with the + fence. + + Please note that the invariant is an implication: e.g. there + can be sel_bb_header_p () insns that don't have a valid LV_SET. + To make an equivalence out of implication we need to invoke + compute_live () after scheduling of an insn that become + sel_bb_header_p () - the overhead will be insignificant because + this case is only possible when we start scheduling of a new + basic block. Also I've just thought about another concerning + issue: + suppose we have a function from a single insn. So far we've + stripped that insn from the stream in move_op () - and, hence, + deleted the only valid LV_SET - how are we supposed to get a + valid LV_SET for the inserted insn out of nowhere? */ - /* Make a move. This call will remove the original operation, - insert all necessary bookkeeping instructions and update the - data sets. After that all we have to do is add the operation - at before BND_TO (BND). */ - b = move_op (BND_TO (bnd), rhs_seq, best_reg, - NULL, NULL, NULL, c_rhs); + { + insn_t prev_insn = PREV_INSN (place_to_insert); + basic_block bb = BLOCK_FOR_INSN (place_to_insert); + basic_block prev_bb = bb->prev_bb; + basic_block next_bb; + + if (!NOTE_INSN_BASIC_BLOCK_P (place_to_insert) + || prev_insn == NULL_RTX + /* Or it is a label, a barrier or something strange + alike. */ + || !INSN_P (prev_insn) + || BLOCK_FOR_INSN (prev_insn) != prev_bb + || !in_current_region_p (prev_bb) + || control_flow_insn_p (prev_insn)) + { + prev_bb = bb; - /* We should be able to find the expression we've chosen for - scheduling. */ - gcc_assert (b); + /* Save new_bb_head to update lv_set on. */ + if (!NOTE_INSN_BASIC_BLOCK_P (place_to_insert) + && !sel_bb_end_p (place_to_insert)) + new_bb_head = NEXT_INSN (place_to_insert); - /* Despite the replacement already done in find_best_rhs_and_reg, - do it once more, because that were done only in local av_vliw - set (we can't do actual replacement there because at the - moment we don't know yet which rhs is the best one. */ - if (best_reg && REGNO (best_reg) != rhs_dest_regno (c_rhs)) + /* Split block to generate a new floating bb header. */ + next_bb = sel_split_block (bb, place_to_insert); + } + else { - static int reg_rename_this_tick = 0; + gcc_assert (single_succ (prev_bb) == bb); - replace_dest_with_reg_in_rhs (c_rhs, best_reg); - - if (HARD_REGISTER_P (best_reg)) - { - reg_rename_tick[REGNO (best_reg)] = ++reg_rename_this_tick; - regs_ever_live[REGNO (best_reg)] = 1; - } + place_to_insert = prev_insn; + next_bb = bb; } - insn = RHS_INSN (c_rhs); - - gcc_assert (!INSN_IN_STREAM_P (insn)); - - /* Find a place for C_RHS to schedule. - We want to have an invariant that only insns that are - bb_header_p () have a valid LV_SET. But, in the same time, - we don't want overhead from recomputation of compute_live () - for the half of a block after each movement. Resolution of - this is floating bb header that will advance along with the - fence. - - Please note that the invariant is an implication: e.g. there - can be bb_header_p () insns that don't have a valid LV_SET. - To make an equivalence out of implication we need to invoke - compute_live () after scheduling of an insn that become - bb_header_p () - the overhead will be insignificant because - this case is only possible when we start scheduling of a new - basic block. Also I've just thought about another concerning - issue: - suppose we have a function from a single insn. So far we've - stripped that insn from the stream in move_op () - and, hence, - deleted the only valid LV_SET - how are we supposed to get a - valid LV_SET for the inserted insn out of nowhere? */ - - { - insn_t prev_insn = PREV_INSN (place_to_insert); - basic_block bb = BLOCK_FOR_INSN (place_to_insert); - basic_block prev_bb = bb->prev_bb; - - gcc_assert (!LV_SET_VALID_P (insn)); - - if (!NOTE_INSN_BASIC_BLOCK_P (place_to_insert)) - { - /* Bookkeeping insn was generated at the end of BB. - Create new floating bb header for it. */ - bb = sel_split_block (bb, place_to_insert); - new_bb_header = bb_head (bb); - } - else if (prev_insn == NULL_RTX - /* Or it is a label, a barrier or something strange - alike. */ - || !INSN_P (prev_insn) - || BLOCK_FOR_INSN (prev_insn) != prev_bb - || !in_current_region_p (prev_bb) - || control_flow_insn_p (prev_insn)) - /* Split block to generate a new floating bb header. */ - bb = sel_split_block (bb, NULL); - else - { - gcc_assert (single_succ (prev_bb) == bb); + if (sel_bb_empty_p (next_bb)) + sel_merge_blocks (prev_bb, next_bb); - place_to_insert = prev_insn; - } + gcc_assert (BLOCK_FOR_INSN (place_to_insert) == prev_bb); - gcc_assert (BLOCK_FOR_INSN (place_to_insert) == bb->prev_bb); + /* Now do some cleanup: remove empty basic blocks after + BB. */ - /* Now do some cleanup: remove empty basic blocks after - BB. */ + next_bb = prev_bb->next_bb; - /* !!! Can't use bb_empty_p here because it returns true on - empty blocks with labels. */ - while (BB_HEAD (bb) == BB_END (bb) - && in_current_region_p (bb)) - { - basic_block next_bb = bb_next_bb (bb); + /* !!! Can't use bb_empty_p here because it returns true on + empty blocks with labels. */ + while (BB_HEAD (next_bb) == BB_END (next_bb) + && in_current_region_p (next_bb)) + { + bb = next_bb->next_bb; - sel_remove_empty_bb (bb, true); - bb = next_bb; - } - } + sel_remove_empty_bb (next_bb, true, true); + next_bb = bb; + } + } - insn_init.what = INSN_INIT_WHAT_INSN; - insn_init.todo = INSN_INIT_TODO_NOTHING; + /* Add the instruction. */ + insn = gen_insn_from_expr_after (c_rhs, seqno, place_to_insert); + clear_expr (c_rhs); - /* Add the instruction. */ - add_insn_after (RHS_INSN (c_rhs), place_to_insert); + ++INSN_SCHED_TIMES (insn); - if (NOTE_INSN_BASIC_BLOCK_P (place_to_insert)) - { - gcc_assert (!new_bb_header); - new_bb_header = insn; - } + if (NOTE_INSN_BASIC_BLOCK_P (place_to_insert)) + { + gcc_assert (new_bb_head == NULL_RTX); + new_bb_head = insn; + } - /* Initialize LV_SET of the bb header. */ - if (new_bb_header) - { + if (new_bb_head != NULL_RTX) + /* Initialize LV_SET of the bb header. */ + { /* !!! TODO: We should replace all occurencies of LV_SET_VALID_P () with LV_SET () != NULL. Overwise it is not clear what a valid and invalid lv set is. */ - if (LV_SET (new_bb_header) == NULL) - LV_SET (new_bb_header) = get_clear_regset_from_pool (); + if (LV_SET (new_bb_head) == NULL) + LV_SET (new_bb_head) = get_clear_regset_from_pool (); - ignore_first = true; - compute_live (new_bb_header); - } - } + ignore_first = true; + compute_live (new_bb_head); + } + } av_set_clear (&rhs_seq); /* Advance the DFA. */ if (recog_memoized (insn) >= 0) { + gcc_assert (!INSN_ASM_P (insn)); + asm_p = false; + memcpy (temp_state, FENCE_STATE (fence), dfa_state_size); succs_n = state_transition (FENCE_STATE (fence), insn); @@ -3189,8 +3624,7 @@ fill_insns (fence_t fence, int seqno, ilist_t **scheduled_insns_tailpp) } else { - asm_p = (GET_CODE (PATTERN (insn)) == ASM_INPUT - || asm_noperands (PATTERN (insn)) >= 0); + asm_p = INSN_ASM_P (insn); /* This could be an ASM insn which we'd like to schedule on the next cycle. */ @@ -3202,11 +3636,8 @@ fill_insns (fence_t fence, int seqno, ilist_t **scheduled_insns_tailpp) pipelining, tick computations etc. */ memcpy (temp_state, FENCE_STATE (fence), dfa_state_size); - INSN_SEQNO (insn) = seqno; - INSN_SCHED_CYCLE (insn) = FENCE_CYCLE (fence); INSN_AFTER_STALL_P (insn) = FENCE_AFTER_STALL_P (fence); - - ++VINSN_SCHED_TIMES (INSN_VI (insn)); + INSN_SCHED_CYCLE (insn) = FENCE_CYCLE (fence); if (asm_p) advance_one_cycle (fence); @@ -3230,8 +3661,7 @@ fill_insns (fence_t fence, int seqno, ilist_t **scheduled_insns_tailpp) line_start (); print ("Scheduling insn: "); - dump_insn_1 (insn, (DUMP_INSN_UID | DUMP_INSN_BBN | DUMP_INSN_PATTERN - | DUMP_INSN_COUNT | DUMP_INSN_CYCLE)); + dump_insn_1 (insn, 1); line_finish (); /* Add new boundaries. */ @@ -3251,6 +3681,14 @@ fill_insns (fence_t fence, int seqno, ilist_t **scheduled_insns_tailpp) bnds_tail1 = *bnds_tailp1; blist_remove (bndsp); + + /* Check invariants. */ + + /* Check that the recent movement didn't destroyed loop + structure. */ + gcc_assert (!flag_sel_sched_pipelining_outer_loops + || current_loop_nest == NULL + || loop_latch_edge (current_loop_nest)); } while (*bndsp != bnds_tail1); @@ -3318,17 +3756,46 @@ fill_insns (fence_t fence, int seqno, ilist_t **scheduled_insns_tailpp) block_finish (); } +/* All exprs in ORIG_OPS must have the same destination register. + Return that register. */ +static rtx +get_dest_reg_from_orig_ops (av_set_t orig_ops) +{ + rtx reg = NULL_RTX; + av_set_iterator av_it; + expr_t expr; + bool first_p = true; + + FOR_EACH_RHS (expr, av_it, orig_ops) + { + rtx x = expr_dest_reg (expr); + + if (first_p) + { + first_p = false; + reg = x; + } + else + gcc_assert (reg == x + || (reg != NULL_RTX && x != NULL_RTX + && REGNO (reg) == REGNO (x))); + } + + return reg; +} + /* Move up the operations from ORIG_OPS set traversing the dag started from INSN. PATH represents the edges traversed so far. REG is the register chosen for scheduling the current rhs. Insert bookkeeping code in the join points. Return the current rhs. */ static bool -move_op (insn_t insn, av_set_t orig_ops, rtx reg, ilist_t path, edge e1, - edge e2, rhs_t c_rhs) +move_op (insn_t insn, av_set_t orig_ops, ilist_t path, edge e1, edge e2, + rhs_t c_rhs) { rhs_t rhs; basic_block bb; bool c_rhs_inited_p; + rtx reg; line_start (); print ("move_op("); @@ -3347,7 +3814,7 @@ move_op (insn_t insn, av_set_t orig_ops, rtx reg, ilist_t path, edge e1, return false; } - orig_ops = av_set_copy (orig_ops, UNIQUE_RHSES); + orig_ops = av_set_copy (orig_ops); /* If we've found valid av set, then filter the orig_ops set. */ if (INSN_AV_VALID_P (insn)) @@ -3369,9 +3836,13 @@ move_op (insn_t insn, av_set_t orig_ops, rtx reg, ilist_t path, edge e1, return false; } - av_set_leave_one (&orig_ops); + /* !!! When using different types of speculation we must not leave + just one element in orig_ops. */ + /*av_set_leave_one (&orig_ops);*/ } + reg = get_dest_reg_from_orig_ops (orig_ops); + /* Look at the insn and decide if it could be an ancestor of currently scheduling operation. If it is so, then the insn "dest = op" could either be replaced with "dest = reg", because REG now holds the result @@ -3385,12 +3856,13 @@ move_op (insn_t insn, av_set_t orig_ops, rtx reg, ilist_t path, edge e1, When traversing the DAG below this insn is finished, insert bookkeeping code, if the insn is a joint point, and remove leftovers. */ - rhs = av_set_lookup_insn (orig_ops, insn); - if (rhs) + rhs = av_set_lookup (orig_ops, INSN_VI (insn)); + + if (rhs != NULL) /* We have found the original operation. Replace it by REG, if it is scheduled as RHS, or just remove it later, if it's an insn. */ { - rhs_copy (c_rhs, rhs); + copy_expr (c_rhs, INSN_EXPR (insn)); c_rhs_inited_p = true; /* When an insn we found is not equal to the insn from the orig_ops @@ -3398,15 +3870,21 @@ move_op (insn_t insn, av_set_t orig_ops, rtx reg, ilist_t path, edge e1, We cannot return original insn in C_RHS because it was already scheduled, and using it would break an assert saying that insn should not be in stream. We need to replace it in C_RHS with - the exact insn that we found. */ - if (INSN_UID (RHS_INSN (c_rhs)) != INSN_UID (insn)) - { - vinsn_t old = RHS_VINSN (c_rhs); - - vinsn_detach (old); - vinsn_attach (INSN_VI (insn)); - RHS_VINSN (c_rhs) = INSN_VI (insn); - } + the exact insn that we found. + + Our infrastructure handles this such case with setting AV_LEVEL () + to -1 in generate_bookkeeping_insn (). + + But when there is identical instructions one of which dominates the + other we have the same assert failing. */ + if (RHS_INSN (c_rhs) != insn) + { + /* We now copy INSN_EXPR (insn) to C_RHS, hence the condition is + always false. */ + gcc_unreachable (); + + change_vinsn_in_expr (c_rhs, INSN_VI (insn)); + } /* For instructions we must immediately remove insn from the stream, so subsequent update_data_sets () won't include this @@ -3415,35 +3893,87 @@ move_op (insn_t insn, av_set_t orig_ops, rtx reg, ilist_t path, edge e1, print ("found original operation!"); - /* If original operation has rhs and the register chosen for that rhs - is not original operation's dest reg, substitute operation's right - hand side with the register chosen. */ - if (RHS_SCHEDULE_AS_RHS (rhs) - && !lhs_equals_reg_p (insn, reg)) + { + insn_t finish_insn = insn; + { - insn_t insn1; - gcc_assert (RHS_SCHEDULE_AS_RHS (rhs)); + ds_t check_ds = get_spec_check_type_for_insn (insn, rhs); - insn1 = replace_src_with_reg (INSN_VI (insn), reg); - transfer_data_sets (insn1, insn); - sched_sel_remove_insn (insn); + if (check_ds != 0) + /* A speculation check should be inserted. */ + { + rtx x; + + x = create_speculation_check_insn_rtx (EXPR_INSN_RTX (rhs), + check_ds); + x = sel_gen_insn_from_rtx_after (x, + INSN_EXPR (finish_insn), + INSN_SEQNO (finish_insn), + finish_insn); + + EXPR_SPEC_DONE_DS (INSN_EXPR (x)) &= ~check_ds; + + finish_insn = x; + + /* If we've generated a data speculation check, make sure + that all the bookkeeping instruction we'll create during + this move_op () will allocate an ALAT entry so that the + check won't fail. + ??? We should ask target if somethings needs to be done + here. */ + /* Commented out due to PR23. */ + /*speculate_expr (c_rhs, ds_get_max_dep_weak (check_ds));*/ + } + else + EXPR_SPEC_DONE_DS (INSN_EXPR (finish_insn)) = 0; - insn = insn1; + gcc_assert (EXPR_SPEC_DONE_DS (INSN_EXPR (finish_insn)) == 0 + && EXPR_SPEC_TO_CHECK_DS (INSN_EXPR (finish_insn)) == 0); } - else + { + rtx cur_reg = expr_dest_reg (c_rhs); + + gcc_assert ((reg == NULL_RTX) == (cur_reg == NULL_RTX)); + + /* If original operation has rhs and the register chosen for + that rhs is not original operation's dest reg, substitute + operation's right hand side with the register chosen. */ + if (reg != NULL_RTX && REGNO (reg) != REGNO (cur_reg)) + { + rtx insn_rtx; + insn_t x; + + replace_dest_with_reg_in_rhs (c_rhs, reg); + + insn_rtx = create_insn_rtx_with_rhs (INSN_VI (insn), reg); + x = sel_gen_insn_from_rtx_after (insn_rtx, + INSN_EXPR (finish_insn), + INSN_SEQNO (finish_insn), + finish_insn); + + finish_insn = x; + } + } + + + if (insn == finish_insn) /* For the insns that don't have rhs just remove insn from the stream. Also remove insn if substituting it's right hand side would result in operation like reg:=reg. This kind of operation is not only excessive, but it may not be supported on certain platforms, e.g. "mov si, si" is invalid on i386. */ - insn_t insn1 = get_nop_from_pool (insn); - - transfer_data_sets (insn1, insn); - + finish_insn = get_nop_from_pool (insn); + + { + insn_t new_start_insn = NEXT_INSN (insn); + + transfer_data_sets (new_start_insn, insn); sched_sel_remove_insn (insn); - insn = insn1; + + insn = new_start_insn; } + } } else { @@ -3452,27 +3982,33 @@ move_op (insn_t insn, av_set_t orig_ops, rtx reg, ilist_t path, edge e1, c_rhs_inited_p = false; - /* ??? Where this comment belongs to? - The sets should be changed before further search, but only - for the real insns. */ - - /* If we're scheduling separate rhs, in order to generate correct code - we need to stop the search at bookkeeping code generated with the - same destination register. This clearing of orig_ops makes - scheduler to choose another destination register for the rhs. - Instructions that are moved as the whole not affected with this - issue because the different insns could be never given the same - register, as it can happen when dealing with separate rhses. */ - /* FIXME: is the presence of reg always equivalent to - RHS_SCHEDULE_AS_RHS? */ - if (lhs_equals_reg_p (insn, reg)) + if (reg != NULL_RTX && lhs_of_insn_equals_to_reg_p (insn, reg)) + /* If we're scheduling separate rhs, in order to generate correct code + we need to stop the search at bookkeeping code generated with the + same destination register. This clearing of orig_ops makes + scheduler to choose another destination register for the rhs. + Instructions that are moved as the whole not affected with this + issue because the different insns could be never given the same + register, as it can happen when dealing with separate rhses. + . + !!! Change the detection of bookkeeping insns to support + clonable but not separable insns. */ av_set_clear (&orig_ops); else { + un_speculate (&orig_ops, insn); + /* Av set ops could have been changed when moving through this insn. To find them below it, we have to un-substitute them. */ - if (flag_sel_sched_substitution) - un_substitute (&orig_ops, insn); + un_substitute (&orig_ops, insn); + } + + /* If all original opernads have been filtered on this branch, + return. */ + if (!orig_ops) + { + block_finish (); + return false; } /* Continue searching. Do recursion here. */ @@ -3481,7 +4017,7 @@ move_op (insn_t insn, av_set_t orig_ops, rtx reg, ilist_t path, edge e1, FOR_EACH_SUCC (succ, succ_i, insn) { bool b; - struct _rhs _x, *x = &_x; + expr_def _x, *x = &_x; if (succ_i.e1) { @@ -3491,26 +4027,46 @@ move_op (insn_t insn, av_set_t orig_ops, rtx reg, ilist_t path, edge e1, && succ_i.e2->dest == BLOCK_FOR_INSN (succ)); } - b = move_op (succ, orig_ops, reg, path, succ_i.e1, succ_i.e2, x); + b = move_op (succ, orig_ops, path, succ_i.e1, succ_i.e2, x); if (b) { + { + enum MOVEUP_RHS_CODE res; + + res = moveup_rhs (x, insn); + gcc_assert (res != MOVEUP_RHS_NULL); + } + if (!c_rhs_inited_p) { - rhs_copy (c_rhs, x); + copy_expr (c_rhs, x); c_rhs_inited_p = true; - rhs_clear (x); } else - rhs_clear (x); + /* We must merge all found expressions to get reasonable + EXPR_SPEC_DONE_DS () for the resulting insn. If we don't + do so then we can first find the expr with epsilon + speculation success probability and only then with the + good probability. As a result the insn will get epsilon + probability and will never be scheduled because of + weakness_cutoff in find_best_rhs_and_reg_that_fits (). + + We also workaround this in can_overcome_dep_p () + that consider low probability speculation success + dependencies as hard ones. + + We call merge_expr_data () here instead of merge_expr () + because due to speculation C_RHS and X may have the + same insns with different speculation types. And as of + now such insns are considered non-correlable. */ + merge_expr_data (c_rhs, x); + + clear_expr (x); } } - - ilist_remove (&path); - /* When insn is substitutable, fix c_rhs by substituting. */ - if (flag_sel_sched_substitution && c_rhs_inited_p) - substitute_rhs (c_rhs, insn); + ilist_remove (&path); } av_set_clear (&orig_ops); @@ -3526,13 +4082,13 @@ move_op (insn_t insn, av_set_t orig_ops, rtx reg, ilist_t path, edge e1, if (e1 && num_preds_gt_1 (insn)) { /* INSN is a joint point, insert bookkeeping code here. */ - bb = generate_bookkeeping_insn (c_rhs, reg, insn, e1, e2); - gcc_assert (bb_header_p (insn)); + bb = generate_bookkeeping_insn (c_rhs, insn, e1, e2); + gcc_assert (sel_bb_header_p (insn)); } else bb = NULL; - if (bb_header_p (insn)) + if (sel_bb_header_p (insn)) { if (AV_LEVEL (insn) == -1) /* This will make assert in update_data_sets () happy. */ @@ -3541,7 +4097,7 @@ move_op (insn_t insn, av_set_t orig_ops, rtx reg, ilist_t path, edge e1, gcc_assert (INSN_AV_VALID_P (insn)); } - if (bb_header_p (insn)) + if (sel_bb_header_p (insn)) { update_data_sets (insn); @@ -3577,7 +4133,7 @@ move_op (insn_t insn, av_set_t orig_ops, rtx reg, ilist_t path, edge e1, gcc_assert (LV_SET_VALID_P (succ)); } - if (bb_header_p (insn)) + if (sel_bb_header_p (insn)) { gcc_assert (LV_SET_VALID_P (insn)); @@ -3651,47 +4207,22 @@ init_seqno_1 (basic_block bb, sbitmap visited_bbs) /* Initialize seqnos for the current region. */ static int -init_seqno (void) +init_seqno (bool rescheduling_p) { sbitmap visited_bbs; visited_bbs = sbitmap_alloc (current_nr_blocks); sbitmap_zero (visited_bbs); - cur_seqno = max_luid - 1; + cur_seqno = sched_max_luid - 1; init_seqno_1 (EBB_FIRST_BB (0), visited_bbs); - /* This assert is not right, when rescheduling code. */ - /*gcc_assert (cur_seqno == 0);*/ - - sbitmap_free (visited_bbs); - - return max_luid - 1; -} - -/* Restore other notes for the whole region. */ -static void -sel_restore_other_notes (void) -{ - int bb; - - for (bb = 0; bb < current_nr_blocks; bb++) - { - basic_block first, last; - first = EBB_FIRST_BB (bb); - last = bb_next_bb (EBB_LAST_BB (bb)); + gcc_assert (rescheduling_p || cur_seqno == 0); - do - { - note_list = BB_NOTE_LIST (first); - restore_other_notes (NULL, first); - BB_NOTE_LIST (first) = NULL_RTX; + sbitmap_free (visited_bbs); - first = bb_next_bb (first); - } - while (first != last); - } + return sched_max_luid - 1; } /* If region is a loop, add an empty basic block before its head, so no @@ -3702,9 +4233,6 @@ add_region_head (void) { basic_block region_head = BASIC_BLOCK (BB_TO_BLOCK (0)); - insn_init.what = INSN_INIT_WHAT_INSN; - insn_init.todo = INSN_INIT_TODO_ALL; - /* We don't want to pipeline weird loops. */ if (EDGE_COUNT (region_head->preds) != 2 || !(EDGE_PRED (region_head, 0)->flags & EDGE_FALLTHRU) @@ -3714,84 +4242,23 @@ add_region_head (void) { basic_block fallthru_pred = EDGE_PRED (region_head, 0)->src; basic_block jump_pred = EDGE_PRED (region_head, 1)->src; + basic_block new_region_head; if (in_current_region_p (fallthru_pred) || !in_current_region_p (jump_pred)) return false; - sel_create_basic_block_before (region_head); - - return true; - } -} - -/* While pipelining outer loops, returns TRUE if BB is a loop preheader. */ -static bool -is_loop_preheader_p (basic_block bb) -{ - /* A preheader may even have the loop depth equal to the depth of - the current loop, when it came from it. Use topological sorting - to get the right information. */ - if (flag_sel_sched_pipelining_outer_loops - && current_loop_nest) - { - struct loop *outer; - /* BB is placed before the header, so, it is a preheader block. */ - if (BLOCK_TO_BB (bb->index) - < BLOCK_TO_BB (current_loop_nest->header->index)) - return true; - - /* Support the situation when the latch block of outer loop - could be from here. */ - for (outer = current_loop_nest->outer; outer; outer = outer->outer) - if (considered_for_pipelining_p (outer) && outer->latch == bb) - gcc_unreachable (); - } - return false; -} - -/* Removes the loop preheader from the current region and saves it in - PREHEADER_BLOCKS of the father loop, so they will be added later to - region that represents an outer loop. - This function is only used with -fsel-sched-pipelining-outer-loops. */ -static void -sel_remove_loop_preheader (void) -{ - int i, old_len; - int cur_rgn = CONTAINING_RGN (BB_TO_BLOCK (0)); - basic_block bb; - VEC(basic_block, heap) *preheader_blocks - = LOOP_PREHEADER_BLOCKS (current_loop_nest->outer); + can_add_real_insns_p = false; + new_region_head = sel_create_basic_block_before (region_head); + can_add_real_insns_p = true; - gcc_assert (flag_sel_sched_pipelining_outer_loops && current_loop_nest); - old_len = VEC_length (basic_block, preheader_blocks); + RGN_WAS_PIPELINED_P (CONTAINING_RGN (BB_TO_BLOCK (0))) = 1; - /* Add blocks that aren't within the current loop to PREHEADER_BLOCKS. */ - for (i = 0; i < RGN_NR_BLOCKS (cur_rgn); i++) - { - bb = BASIC_BLOCK (BB_TO_BLOCK (i)); + glat_start[new_region_head->index] = glat_start[region_head->index]; + glat_start[region_head->index] = NULL; - /* If the basic block belongs to region, but doesn't belong to - corresponding loop, then it should be a preheader. */ - if (is_loop_preheader_p (bb)) - VEC_safe_push (basic_block, heap, preheader_blocks, bb); - } - - /* Remove these blocks only after iterating over the whole region. */ - for (i = VEC_length (basic_block, preheader_blocks) - 1; - i >= old_len; - i--) - { - bb = VEC_index (basic_block, preheader_blocks, i); - sel_add_or_remove_bb_1 (bb, false); + return true; } - - if (!considered_for_pipelining_p (current_loop_nest->outer)) - /* Immediately create new region from preheader. */ - make_region_from_loop_preheader (&preheader_blocks); - else - /* Store preheader within the father's loop structure. */ - SET_LOOP_PREHEADER_BLOCKS (current_loop_nest->outer, preheader_blocks); } /* Split all edges incoming to current region, but not those that @@ -3858,12 +4325,9 @@ static bool sel_region_init (int rgn) { int i; - sbitmap bbs; + bb_vec_t bbs; - /* Use *preinit* here, because we want to defer actual initialization - until sched_data_update. */ - if (sched_rgn_local_preinit (rgn)) - return true; + rgn_setup_region (rgn); /* If this loop has any saved loop preheaders from nested loops, add these basic blocks to the current region. */ @@ -3881,36 +4345,66 @@ sel_region_init (int rgn) } } - /* We don't need the semantics of moveup_set_path, because filtering of - dependencies inside a sched group is handled by tick_check_p and - the target. */ - enable_moveup_set_path_p = 0; + bbs = VEC_alloc (basic_block, heap, current_nr_blocks); - /* We need to treat insns as RHSes only when renaming is enabled. */ - enable_schedule_as_rhs_p = (flag_sel_sched_renaming != 0); + for (i = 0; i < current_nr_blocks; i++) + VEC_quick_push (basic_block, bbs, BASIC_BLOCK (BB_TO_BLOCK (i))); - bookkeeping_p = (flag_sel_sched_bookkeeping != 0); - pipelining_p = bookkeeping_p && (flag_sel_sched_pipelining != 0); + sel_init_bbs (bbs, NULL); - bbs = sbitmap_alloc (last_basic_block); - sbitmap_zero (bbs); - for (i = 0; i < current_nr_blocks; i++) - SET_BIT (bbs, BB_TO_BLOCK (i)); + /* Initialize luids and dependence analysis which both sel-sched and haifa + need. */ + sched_init_luids (bbs, NULL, NULL, NULL); + sched_deps_local_init (false); - insn_init.what = INSN_INIT_WHAT_INSN; - insn_init.todo = ((INSN_INIT_TODO_GLOBAL | INSN_INIT_TODO_ALL) - & ~INSN_INIT_TODO_SEQNO); - - init_deps_global (); - sched_data_update (bbs, NULL, NULL); - finish_deps_global (); - sbitmap_free (bbs); + /* Initialize haifa data. */ + { + rgn_setup_sched_infos (); + haifa_init_h_i_d (bbs, NULL, NULL, NULL); + } - /* We were supposed to bail out earlier when this region should not - be scheduled. */ - if (sched_rgn_local_init (rgn, false)) + if (sched_rgn_local_preinit (rgn)) return true; + if (sched_verbose >= 5) + debug_dependencies (); + + /* Compute insn priorities in haifa style. Then free haifa style + dependencies that we've calculated for this. */ + compute_priorities (); + + init_deps_global (); + + /* Setup flags that might affect initialization. */ + { + /* We don't need the semantics of moveup_set_path, because filtering of + dependencies inside a sched group is handled by tick_check_p and + the target. */ + enable_moveup_set_path_p = 0; + + /* We need to treat insns as RHSes only when renaming is enabled. */ + enable_schedule_as_rhs_p = (flag_sel_sched_renaming != 0); + + bookkeeping_p = (flag_sel_sched_bookkeeping != 0); + + pipelining_p = bookkeeping_p && (flag_sel_sched_pipelining != 0); + + reset_sched_cycles_p = (pipelining_p + && !flag_sel_sched_reschedule_pipelined); + } + + sel_setup_sched_infos (); + + /* Main initialization. */ + sel_init_global_and_expr (bbs); + + /* Finalize haifa-specific data. */ + { + haifa_finish_h_i_d (); + } + + VEC_free (basic_block, heap, bbs); + /* Set hooks so that no newly generated insn will go out unnoticed. */ old_rtl_hooks = rtl_hooks; rtl_hooks = sel_rtl_hooks; @@ -3922,49 +4416,40 @@ sel_region_init (int rgn) old_delete_basic_block = rtl_cfg_hooks.delete_basic_block; rtl_cfg_hooks.delete_basic_block = rtl_delete_block_not_barriers; - /* Compute insn priorities in haifa style. Then free haifa style - dependencies that we've calculated for this. */ - compute_priorities (); - - /* !!! We call target.sched.md_init () for the whole region, but we invoke - targetm.sched.md_finish () for every ebb. */ - if (targetm.sched.md_init) - /* None of the arguments are actually used in any target. */ - targetm.sched.md_init (sched_dump, sched_verbose, -1); - - if (sched_verbose >= 5) - debug_dependencies (); - free_rgn_deps (false); - - init_deps_global (); - - first_emitted_uid = sel_max_uid; - - /* Reset register allocation ticks array. */ - memset (reg_rename_tick, 0, sizeof reg_rename_tick); - if (pipelining_p) { /* If pipelining of outer loops is enabled, the loop header is already created with loop optimizer, so if current region has a corresponding loop nest, we should pipeline it. */ if (flag_sel_sched_pipelining_outer_loops) - { - pipelining_p = (current_loop_nest != NULL); - - if (pipelining_p) - split_edges_incoming_to_rgn (); + { + pipelining_p = (current_loop_nest != NULL); - } + if (pipelining_p) + split_edges_incoming_to_rgn (); + } else pipelining_p = add_region_head (); } setup_dump_cfg_params (pipelining_p); + /* !!! We call target.sched.md_init () for the whole region, but we invoke + targetm.sched.md_finish () for every ebb. */ + if (targetm.sched.md_init) + /* None of the arguments are actually used in any target. */ + targetm.sched.md_init (sched_dump, sched_verbose, -1); + + first_emitted_uid = get_max_uid () + 1; + + /* Reset register allocation ticks array. */ + memset (reg_rename_tick, 0, sizeof reg_rename_tick); + bitmap_initialize (forced_ebb_heads, 0); bitmap_clear (forced_ebb_heads); + setup_empty_vinsn (); + return false; } @@ -3973,57 +4458,12 @@ static void sel_region_finish (void) { int i; - struct insn_init_how_deferred_def *p; - - /* FIXME: turn this cleaning up into a function. */ - p = &insn_init.how_deferred; - gcc_assert (p->n == 0); - free (p->insns); - p->insns = NULL; - p->size = 0; - - insn_init.what = INSN_INIT_WHAT_INSN; - insn_init.todo = INSN_INIT_TODO_NOTHING; - - finish_deps_global (); - free_rgn_deps (true); - - /* Selective scheduling freeing of saved of registers uses/sets/clobbers - information. */ - for (i = 0; i < current_nr_blocks; i++) - { - basic_block bb = EBB_FIRST_BB (i); - rtx insn = NEXT_INSN (bb_note (bb)), next_tail = NEXT_INSN (BB_END (bb)); - gcc_assert (EBB_LAST_BB (i) == bb); + sel_finish_new_insns (); - for (; insn != next_tail; insn = NEXT_INSN (insn)) - { - gcc_assert (INSN_P (insn)); - - av_set_clear (&AV_SET (insn)); - } - } - - /* Now we can delete all vinsns. */ - for (i = 0; i < current_nr_blocks; i++) - { - basic_block bb = EBB_FIRST_BB (i); - rtx insn = NEXT_INSN (bb_note (bb)), next_tail = NEXT_INSN (BB_END (bb)); - - gcc_assert (EBB_LAST_BB (i) == bb); - - for (; insn != next_tail; insn = NEXT_INSN (insn)) - { - vinsn_t vi = INSN_VI (insn); + sched_finish_ready_list (); - gcc_assert (VINSN_COUNT (vi) == 1); - - INSN_FINAL_SCHED_CYCLE (insn) = VINSN_SCHED_CYCLE (vi); - - vinsn_detach (vi); - } - } + free_nop_pool (); /* Free the sort vector. */ if (vec_av_set) @@ -4050,7 +4490,7 @@ sel_region_finish (void) /* While pipelining outer loops, skip bundling for loop preheaders. Those will be rescheduled in the outer loop. */ - if (is_loop_preheader_p (bb)) + if (sel_is_loop_preheader_p (bb)) continue; line_start (); @@ -4076,7 +4516,7 @@ sel_region_finish (void) current_sched_info->prev_head = PREV_INSN (head); current_sched_info->next_tail = NEXT_INSN (tail); - if (pipelining_p && !flag_sel_sched_reschedule_pipelined) + if (reset_sched_cycles_p) { int last_clock = 0; int haifa_last_clock = -1; @@ -4105,15 +4545,14 @@ sel_region_finish (void) continue; asm_p = false; - clock = INSN_FINAL_SCHED_CYCLE (insn); + clock = INSN_SCHED_CYCLE (insn); cost = clock - last_clock; /* Initialize HAIFA_COST. */ if (recog_memoized (insn) < 0) { - asm_p = (GET_CODE (PATTERN (insn)) == ASM_INPUT - || asm_noperands (PATTERN (insn)) >= 0); + asm_p = INSN_ASM_P (insn); if (asm_p) /* This is asm insn which *had* to be scheduled first @@ -4184,7 +4623,7 @@ sel_region_finish (void) targetm.sched.variable_issue (sched_dump, sched_verbose, insn, 0); - INSN_FINAL_SCHED_CYCLE (insn) = haifa_clock; + INSN_SCHED_CYCLE (insn) = haifa_clock; last_clock = clock; haifa_last_clock = haifa_clock; @@ -4211,16 +4650,14 @@ sel_region_finish (void) continue; } - clock = INSN_FINAL_SCHED_CYCLE (insn); + clock = INSN_SCHED_CYCLE (insn); cost = (last_clock == -1) ? 1 : clock - last_clock; gcc_assert (cost >= 0); line_start (); - print ("cost: %d\tcycle: %d\t", - cost, INSN_FINAL_SCHED_CYCLE (insn)); - dump_insn_1 (insn, (DUMP_INSN_UID | DUMP_INSN_BBN - | DUMP_INSN_PATTERN)); + print ("cost: %d\t", cost); + dump_insn_1 (insn, 1); line_finish (); if (issue_rate > 1 @@ -4240,17 +4677,15 @@ sel_region_finish (void) { /* md_finish () can possibly emit new insns. Move LV_SETs to ones that happen to be emitted on bb header. */ - int old_todo = insn_init.todo; - - insn_init.todo |= (INSN_INIT_TODO_MOVE_LV_SET_IF_BB_HEADER - /* We don't really have to provide luids for - those insns, but that is faster to - code. */ - | INSN_INIT_TODO_LUID); - + insn_init.what = INSN_INIT_WHAT_INSN; targetm.sched.md_finish (sched_dump, sched_verbose); - insn_init.todo = old_todo; + /* Extend luids so that insns generated by the target will + get zero luid. */ + sched_init_luids (NULL, NULL, NULL, NULL); + + insn_init.todo = INSN_INIT_TODO_MOVE_LV_SET_IF_BB_HEADER; + sel_init_new_insns (); } } @@ -4258,25 +4693,25 @@ sel_region_finish (void) BITMAP_FREE (scheduled_blocks); } + sel_finish_global_and_expr (); + bitmap_clear (forced_ebb_heads); - sel_restore_other_notes (); + free_empty_vinsn (); + + finish_deps_global (); + sched_deps_local_finish (); + sched_finish_luids (); - sched_data_finish (); + sel_finish_bbs (); rtl_cfg_hooks.delete_basic_block = old_delete_basic_block; rtl_cfg_hooks.create_basic_block = old_create_basic_block; rtl_hooks = old_rtl_hooks; - sched_rgn_local_finish (); - - /* Reset OLD_READY_VECLEN. */ - old_ready_veclen = 0; - - /* Remove current loop preheader from this loop. */ - if (flag_sel_sched_pipelining_outer_loops && current_loop_nest) - sel_remove_loop_preheader (); + /* Reset MAX_ISSUE_SIZE. */ + max_issue_size = 0; } @@ -4462,7 +4897,7 @@ sel_sched_region_1 (void) { struct sel_sched_region_2_data_def _data, *data = &_data; - data->orig_max_seqno = init_seqno (); + data->orig_max_seqno = init_seqno (false); if (data->orig_max_seqno < 1) { @@ -4511,11 +4946,12 @@ sel_sched_region_1 (void) for (i = 0; i < current_nr_blocks; i++) { bb = EBB_FIRST_BB (i); - head = bb_head (bb); + head = sel_bb_header (bb); /* While pipelining outer loops, skip bundling for loop - preheaders. Those will be rescheduled in the outer loop. */ - if (is_loop_preheader_p (bb)) + preheaders. Those will be rescheduled in the outer + loop. */ + if (sel_is_loop_preheader_p (bb)) { clear_outdated_rtx_info (bb); continue; @@ -4552,9 +4988,9 @@ sel_sched_region_1 (void) /* Schedule region pre-header first, if not pipelining outer loops. */ bb = EBB_FIRST_BB (0); - head = bb_head (bb); + head = sel_bb_header (bb); - if (is_loop_preheader_p (bb)) + if (sel_is_loop_preheader_p (bb)) /* Don't leave old flags on insns in bb. */ clear_outdated_rtx_info (bb); else if (head != NULL_RTX) @@ -4579,6 +5015,9 @@ sel_sched_region_1 (void) /* Reschedule pipelined code without pipelining. */ loop_entry = EBB_FIRST_BB (1); + /* ??? Why don't we assert that EBB_FIRST_BB (1) is an + actual loop entry? There must be something wrong if we + somehow created an extra block before the loop. */ while (loop_entry && EDGE_COUNT (loop_entry->preds) == 1) loop_entry = loop_entry->next_bb; @@ -4602,13 +5041,20 @@ sel_sched_region_1 (void) insn = NEXT_INSN (insn)) { gcc_assert (INSN_P (insn)); + INSN_AFTER_STALL_P (insn) = 0; INSN_SCHED_CYCLE (insn) = 0; - VINSN_SCHED_TIMES (INSN_VI (insn)) = 0; - INSN_FINAL_SCHED_CYCLE (insn) = 0; + + /* ??? Should we reset those counters which reside in + INSN_EXPR field (e.g. SPEC and SCHED_TIMES)? */ + /* For now we do need to zero SCHED_TIMES because we don't + want to skip dependencies from any instruction. This + will be a subject to consider when we implement better + dependency tracking. */ + INSN_SCHED_TIMES (insn) = 0; } } - data->orig_max_seqno = init_seqno (); + data->orig_max_seqno = init_seqno (true); flist_tail_init (new_fences); /* Mark BB as head of the new ebb. */ @@ -4649,17 +5095,17 @@ sel_sched_region (int rgn) region_start = PARAM_VALUE (PARAM_REGION_START); region_stop = PARAM_VALUE (PARAM_REGION_STOP); region_p = (PARAM_VALUE (PARAM_REGION_P) == 1); - + if (region_p) schedule_p = (region_start <= region) && (region <= region_stop); else schedule_p = (region_start > region) || (region > region_stop); - + if (schedule_p) sel_sched_region_1 (); else - /* This will set force initialization of INSN_FINAL_SCHED_CYCLEs. */ - pipelining_p = true; + /* Force initialization of INSN_SCHED_CYCLEs for correct bundling. */ + reset_sched_cycles_p = true; } sel_region_finish (); @@ -4680,17 +5126,50 @@ sel_global_init (void) pipeline_outer_loops_init (); setup_sched_dump_to_stderr (); - setup_sched_and_deps_infos (); + + /* Setup the only info sched_init () needs. */ + sel_setup_common_sched_info (); /* We want to create new pseudos occasionally. */ if (!reload_completed) no_new_pseudos = 0; sched_init (); + + /* Init lv_sets. */ + { + bb_vec_t bbs = VEC_alloc (basic_block, heap, n_basic_blocks); + basic_block bb; + + FOR_ALL_BB (bb) + VEC_quick_push (basic_block, bbs, bb); + + sched_init_bbs (bbs, NULL); + + VEC_free (basic_block, heap, bbs); + } + sched_rgn_init (flag_sel_sched_single_block_regions != 0, flag_sel_sched_ebb_regions != 0); + /* Init the set of all registers available in the function. */ + INIT_REG_SET (sel_all_regs); + + { + int i; + + for (i = 0; i < FIRST_PSEUDO_REGISTER; i++) + { + if (regs_ever_live[i] + || call_used_regs[i]) + SET_REGNO_REG_SET (sel_all_regs, i); + } + } + setup_nop_and_exit_insns (); + + sel_extend_insn_rtx_data (); + init_lv_sets (); } @@ -4698,30 +5177,18 @@ sel_global_init (void) static void sel_global_finish (void) { - basic_block bb; - - FOR_EACH_BB (bb) - { - insn_t head; - insn_t next_tail; + free_lv_sets (); - get_ebb_head_tail (bb, bb, &head, &next_tail); - next_tail = NEXT_INSN (next_tail); + sel_finish_insn_rtx_data (); - /* We should scan through all the insns because bundling could - have emitted new insns at the bb headers. */ - while (head != next_tail) - { - release_lv_set_for_insn (head); - head = NEXT_INSN (head); - } - } + free_nop_and_exit_insns (); - free_exit_insn_data (); - free_lv_sets (); free_regset_pool (); + CLEAR_REG_SET (sel_all_regs); + sched_rgn_finish (); + sched_finish_bbs (); sched_finish (); if (!reload_completed) @@ -4756,8 +5223,11 @@ selective_scheduling_run (void) if (sel_sched_verbose) sched_verbose_param = sel_sched_verbose; + setup_dump_cfg_params (false); + sel_dump_cfg_1 ("before-init", - SEL_DUMP_CFG_BB_INSNS | SEL_DUMP_CFG_FUNCTION_NAME); + (SEL_DUMP_CFG_BB_INSNS | SEL_DUMP_CFG_FUNCTION_NAME + | SEL_DUMP_CFG_BB_LIVE)); sel_global_init (); @@ -4788,7 +5258,8 @@ selective_scheduling_run (void) sel_global_finish (); sel_dump_cfg_1 ("after-finish", - SEL_DUMP_CFG_BB_INSNS | SEL_DUMP_CFG_FUNCTION_NAME); + (SEL_DUMP_CFG_BB_INSNS | SEL_DUMP_CFG_FUNCTION_NAME + | SEL_DUMP_CFG_BB_LIVE)); sched_verbose_param = old_sched_verbose_param; } @@ -4808,6 +5279,8 @@ gate_handle_sel_sched (void) #endif } +static int sel1_run = 0; + /* Run instruction scheduler. */ static unsigned int handle_sel_sched (void) @@ -4815,10 +5288,27 @@ handle_sel_sched (void) if (reload_completed) split_all_insns (1); #ifdef INSN_SCHEDULING - if (flag_selective_scheduling || flag_selective_scheduling2) - selective_scheduling_run (); - else - schedule_insns (); + { + int now; + int start; + int stop; + bool do_p; + + now = ++sel1_run; + start = PARAM_VALUE (PARAM_SEL1_START); + stop = PARAM_VALUE (PARAM_SEL1_STOP); + do_p = (PARAM_VALUE (PARAM_SEL1_P) == 1); + + if (do_p) + do_p = (start <= now) && (now <= stop); + else + do_p = (start > now) || (now > stop); + + if ((flag_selective_scheduling || flag_selective_scheduling2) && do_p) + selective_scheduling_run (); + else + schedule_insns (); + } #endif return 0; } diff --git a/gcc/target-def.h b/gcc/target-def.h index fa6067b1a80..ab483482f22 100644 --- a/gcc/target-def.h +++ b/gcc/target-def.h @@ -317,9 +317,12 @@ Foundation, 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. #define TARGET_SCHED_FREE_SCHED_CONTEXT 0 #define TARGET_SCHED_SPECULATE_INSN 0 #define TARGET_SCHED_NEEDS_BLOCK_P 0 -#define TARGET_SCHED_GEN_CHECK 0 +#define TARGET_SCHED_GEN_SPEC_CHECK 0 #define TARGET_SCHED_FIRST_CYCLE_MULTIPASS_DFA_LOOKAHEAD_GUARD_SPEC 0 #define TARGET_SCHED_SET_SCHED_FLAGS 0 +#define TARGET_SCHED_GET_INSN_SPEC_DS 0 +#define TARGET_SCHED_GET_INSN_CHECKED_DS 0 +#define TARGET_SCHED_SKIP_RTX_P 0 #define TARGET_SCHED \ @@ -351,9 +354,12 @@ Foundation, 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. TARGET_SCHED_FREE_SCHED_CONTEXT, \ TARGET_SCHED_SPECULATE_INSN, \ TARGET_SCHED_NEEDS_BLOCK_P, \ - TARGET_SCHED_GEN_CHECK, \ + TARGET_SCHED_GEN_SPEC_CHECK, \ TARGET_SCHED_FIRST_CYCLE_MULTIPASS_DFA_LOOKAHEAD_GUARD_SPEC, \ - TARGET_SCHED_SET_SCHED_FLAGS} + TARGET_SCHED_SET_SCHED_FLAGS, \ + TARGET_SCHED_GET_INSN_SPEC_DS, \ + TARGET_SCHED_GET_INSN_CHECKED_DS, \ + TARGET_SCHED_SKIP_RTX_P} #define TARGET_VECTORIZE_BUILTIN_MASK_FOR_LOAD 0 #define TARGET_VECTORIZE_BUILTIN_VECTORIZED_FUNCTION default_builtin_vectorized_function diff --git a/gcc/target.h b/gcc/target.h index 3543bf1902a..4ddb56b127b 100644 --- a/gcc/target.h +++ b/gcc/target.h @@ -375,9 +375,9 @@ struct gcc_target /* The following member value is a pointer to a function called by the insn scheduler. It should return true if the check instruction - corresponding to the instruction passed as the parameter needs a + corresponding to speculation type passed as the parameter needs a recovery block. */ - bool (* needs_block_p) (rtx); + bool (* needs_block_p) (int); /* The following member value is a pointer to a function called by the insn scheduler. It should return a pattern for the check @@ -387,7 +387,7 @@ struct gcc_target simple check). If the mutation of the check is requested (e.g. from ld.c to chk.a), the third parameter is true - in this case the first parameter is the previous check. */ - rtx (* gen_check) (rtx, rtx, bool); + rtx (* gen_spec_check) (rtx, rtx, int); /* The following member value is a pointer to a function controlling what insns from the ready insn queue will be considered for the @@ -401,6 +401,17 @@ struct gcc_target information about the speculation capabilities of the target. The parameter is a pointer to spec_info variable. */ void (* set_sched_flags) (struct spec_info_def *); + + /* Return speculation types of the instruction passed as the parameter. */ + int (* get_insn_spec_ds) (rtx); + + /* Return speculation types that are checked for the instruction passed as + the parameter. */ + int (* get_insn_checked_ds) (rtx); + + /* Return bool if rtx scanning should just skip current layer and + advance to the inner rtxes. */ + bool (* skip_rtx_p) (rtx); } sched; /* Functions relating to vectorization. */ |